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In 1991, the U.S. Departmem of Labor funded 
Appropriate Solutions, inc. (ASI) to carry out a 
technical assistance project on JTPA participant 
follow-up. 


The first phase of the project involved visits to a 
number of States and SDAs. These visits had two 
purposes: 1) to assist the organizations visited; and 
2) to gather information on the data collection and 
utilization problems being experienced in the job 
training system. 


The sites represented the diversity within the JTPA 
system. in some of the States visited, each SDA 
collected its own data. Other States had centralized 
systems and contracted with a survey organization 
for data collection services. One State had a 
centralized system, but interviews were being 
conducted in-house. In spite of these differences, 
sites tended to experience similar problems and were 
concerned about the same issues. 


The second phase of the project was to prepare a 
supplement te the 1990 Technical Assistance Guide 
(1980 TAG.) on follow-up which would not only 
presem new information, but would also enhance the 
accessibility and utility of the material in the 1990 
TAG. 


Instead of producing a single document, i was 
decided to divide the supplement into twelve parts 
that are sensitive to the research and planning cycles 
of States and SDAs. 


The supplement has been named T.AG.*, which 
tells the reader that this publication Is intended to go 
beyond the 1990 TAG. and provide value-added 
information. Its twelve modules will present materials 


FLAGS T.A.G.* is a twelve module supplement to the 1990 Technicai Assistance Guide for JTPA Follow-Up and 
Validation. To better meet our objectives, we have adopted two conventions. 


You will find that we will consistently “key” you to relevant information in the 1990 T.AG. and “flag* related matorial 


in other T.A.G.* Modules with the following symbols. 


T.AG.: pp. 
a 


Information in the 1990 T.A.G. will be “keyed” by this 
symbol. 


a 
ee 


information in the T.A.G.* Modules will be “flagged” 
by this symbol. 
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The primary emphasis of T.A.G.* will be on helping 
States and SDAs get the most cut of their follow-up 
and evaluation systems. TAG.* should be 
circulated and discussed among the people who 
contribute to making follow-up work at all levels. We 
are dealing with serious material, but we hope to 
present ht in a fashion that you will not only find 


helpful, but enjoyable. 


The original 1990 T.A.G. is an essential reference for 
people involved in follow-up, and we will refer to it 
often. 


T.AG.* should be disseminated to anyone and 
everyone who wants to know more about follow-up, 
postprogram evaluation, performance management, 
and policy analysis. information conceming how to 
request additional copies of the 1990 TAG. and 
T.A.G.* modules can be found on page & We 
encourage wide Gissemination. 3 3 


| DEFINING THE JTPA SYSTEM 


Regardless of how data collection is organized within 
a State, JTPA participant follow-up is a complex 
system. When any of the elements which make up 
that system do not operate effectively, or when 
information does not flow smoothly between the 
elements, the entire system is less effective. 


The elements which comprise follow-up systems are 
people who fill specialized roles within their 
organizations. In sinaller SDAs, one person may fill 
several roles. Typically, however, follow-up involves 
many people, often in more than one organization. It 
is not surprising, therefore, that communication and 
coordinaiion issues were important to the sites 
visited by the Foliow-up Technical Assistance project. 


States and SDAs that want to strengthen their 
programs may want to consider the following 
questions as @ starting point for defining their 
systems more Clearly: 


B Who is included in the follow-up system? 
What do they do? What resources do they 
need to carry out their functions? Oata 
collection is only one part of the follow-up 
system. Follow-up begins as counselors and 
instructors collect contact information and 
encourage participants to cooperate. it Goes 
not end until the information collected is 
used by decision makers. 


a What are the connections between 
members of the system? That is, how 


does work end information flow? Follow- 
up is an orderly process, involving a 
definable sequence of events. When steps in 
that process do not occur, occur late, or are 
carried out badly, the entire system suffers. 


a What kinds of feedback are available to 
the members of the system? Feedback is 
imponant because it enables the system to 
adjust Rself to improve Rs performance. 
Follow-up systems can generate lots of 
information about their own functioning, but 
feedback Is rot effective unless it is received 
and understood by someone who has the 
capacity and volition to respond. 


a Where are authority, responsibility, and 
accountability located within the system? 
Typically, ditieremt people are responsible for 
Gifleremt stages of the follow-up process, but 
there is also a need for someone who is 
responsible for the system as a whole. in 
either case, Ff & is not clear where the 
authority to make specific decisions is 
located, decisions will not be made; and the 
status quo will be perpetuated. 


The diagrarn on the next page shows that the system 
has a great deal of interaction, and, # operating 
properly, a great amount of communication. There 
are no unimportamt parts in the system. Because 
everyone in this systen) has an inherent inerest in 
the success of the JTP/ program. they also have an 


X 
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interest in the quality of the follow-up and 

postprogram evaluation. renowenTs 

The audiences at the bottom of the diagram are ra \ 

many and include USDOL, the Governor, Congress, COUNSELORS, 

the media, and the general public. Each eleme:tt in aw -_ 

the diagram is a potential user of follow-up 

information. They are also potentia) advisers to o her 

parts of the system. Ciear communication between PROGRAM ¢ > wey 
all elements leads to better and more responsive _ ya 
programs. 6 3 al 


ideally, data collection and utilization go hand in 

hand. Organizations collect what they need and use JTPA follow-up systems have generally been 
what they collect. Too often, however, the data designed to meet federal requirements for data 
collection process is designed without much thought collection and reporting. At the same time, States 


as to how the data collected will be used. When that and SDAs are discovering that those systems do not 
happens, the potential value of the data decreases always fully meet their own informational needs. 
dramatically. T.A.G.* is intended to address these problems. 

The three prerequisites to utilization are content, We view the issue of utilization in terms of three 


quality, and access. Although collecting data which levels. These levels represert increasing 
are relevant to our information needs is essential to sophistication in using information in performance 
utilization, it is not enough by itself. We cannot use management and policy analysis. Each level of 
what we collect unless we are confident that what it utilization builds and expands upon the 
telis us is valid. And, regarciess of content and level. 

(quality, we cannot use our data unless we can get i 

when we need it, the way we need it. 


LEVEL 3: Program evaluation and policy analysis, including 
support of longer term planning and policy development 
and measurement of program effectiveness. 


af program strengths and weaknesses and continuous 
feedback on the quality of the placements made. 


LEVEL 1: Compliance with federal requirements including 
annual reporting and implementation of performance 
standards. 


TAG.* 


Regardiess of your Local or State resources, we 
believe that everybody is capable of working at Level 
2 and Level 3 if that is what they want. TA.G.* wil 
show you how. 


Here are some key things to remember. 


B if you don't do anything with the data except 
fill in torms, then foliow-up Is @ cost. 


S When you go beyond calculating mandatory 
performance measures, and begin to use the 
data in performance management and policy 
analysis, then follow-up becomes an 
investment. 
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a Data is not Power; it's just data. When you 
use Data, f becomes information. When you 
study the information, & becomes 
Knowledge. When you ieam from that 


Knowledge, k& becomes Wisdom. 
information, Knowledge and Wisdom are 
Power. 
TAG.: pp. 13-15. 
a 


Utilization of your data will be a recurrent theme in 
T.AG.*. We wil be looking at all aspects of the 
follow-up and survey process to provide you with 
tools and guidance to help you achieve the level of 
utilization that you desire. That process begins with 
your survey design. % 3 


“As long as | have you on the phone..." 
A shorn GUIDE TO QUESTIONNAIRE DESIGN 


Follow-up is a required activity for collecting 
postprogram data that are used in calculating 
performance measures and in awarding incentives. 


The major cost in acquiring follow-up data lies in 
locating the respondents and getting them on the 
telephone. For a modest additiona! cost, States and 
SDAs can add other questions which will help them 
understand what is happening in their programs and 
with their participants. This is not as simple as it 
sounds. interviewing is more than just asking 
questions. Likewise, designing a questionnaire ts 
much more than just writing questions. 


TAG.: pp. 8-9. 
a ee 


ANALYSIS STARTS with DESIGN 


The first task is io determine what you want to know 
ard what you are going to do with the answers. 
What is this going to tell us? Will this be of help to 
State or SDA program p.anning and design? Will this 
help us identify gaps in our program or needs we 
may have overlooked? 


There are many good reasons to ask additional 


questions. The important thing is to have a reason 
for each question asked. 


Avoid the common trap of asking too many questions 
just because “i would be nice to know.” If you never 
get around to analyzing your data, you have wasted 
time and resources. 


The preferred method for follow-up surveys is 
telephone interviewing. Because you will be asking 
the questions orally, you must listen to how they 
sound — not just in your head as you read them, but 
aloud. Your design is not finished unti you are 
satisfied that the questionnaire sounds right and does 
what you want. You are after clear and unarnbiguous 
communication. 


ONE THOUGHT 


A well Gesigned question contains a single thought. 
You may be tempted to construct a question with 
very complicated instructions or multiple thoughts. 
Keep 8 simple. Understand what you are asking and 


TAG.* — Module 1 — Page 5 


what the respondent is answering Consider the 
following: 


How ‘would you rate the instructors 
and training facility — excellent, good, 
fair or poor? 


How do respondents answer ff they think the 
instructors were superb, but were cursed with a 
faciity that should have been razed in the 1800's? 
You cannot design good questions unless you can 
look at them through the respondent's eyes. The 
above example should have been two questions: 


How would you rate the instructors — 
excellent, good, fair or poor? 

How would you rate the training 
facility - excellent, good, fair or poor? 


Although you now have two questions instead of one, 
the answers are irierpretable. The two questions will 
probably go faster than the one because the 
interviewer will not be stopped by respondents 
asking for clarification or qualifying their responses. 


One CLEAR THOUGHT 


Yes, t was clear to you. No, it was not clear to 
anybody else. As you design questions, you fall into 
mind sets. The following is a real example of a failure 
of this kind. 


There are a lot of things people can 
do when they are looking for a job. if 
you were looking for a job, what are 
the 2 of 3 most effective things you 
would do to find a job? 


The designers of that question thou:/t they would 
find things such as checking help wanted ads, 
registering with an employment agency, etc. 


The respondents, however, looked at it differently 
and related things they did such as buying a sult and 
getting a haircut. This is a better question. 


There are a lot of things people can 
do when they are looking for a job. if 
you were looking for a job, what do 


you feel are the 2 or 3 best sources of 
information about new jobs? 


Onder Et FECT 


Writing a good question is only one element in the art 
of design. The order in which you put those 
questions will also affect the accuracy of the data you 
collect. 


All things being equal, your questionnaire should 
move from the general to the specific. A broad, 
open-ended question should precede specific close- 
ended questions to prevemt suggesting answers to 
respondents. 


An open-ended question has 10 specific categories 
given for responses. An ex mople is the above 
Question on sources for employn °nt leads. A close- 
ended question requires a specific response, such as 
the question on evaluating instructors. 


A high school student in Columbus, Ohio did a 
science fair experiment on order effects in 1969. He 
asked all of the Freshman class two sets of 
questions. For half of the group he asked a general 
question about problems facing a high school 
student today followed by three specific problems. 
For the other half, he reversed the order asking the 
specific questions first. 


When the data were analyzed, the second group, 
which was asked specific before general, 
disproportionately gave one of the three specific 
problems as a response to the general question. The 
respondent had been “cued” to possible respcnses 
by hearing the specific questions first. Opinions had 
been created. 


category (usually 9 or blank) should also be included 


S 
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in non-mandatory questions so you can capture 
these responses. 


LEVEL OF DIFFICULTY 


Your respondents should be able to answer every 
question truthfully and accurately. if you write a 
question so complicated that not even the survey 
design staff could give an accurate answer, then you 
have a problem. Consider the following two 
questions taken from one State's survey. 


Ir what month and year were you last 
enrolied full time in school not 


counting a JTPA Program? 


How many of the years since that time 
have you worked at least 6 months? 


For some persons, the first question is very difficult. 
The second question is a “survey kiter which can 
bring the interview to a dead stop. 


Adult participants typically range from 22 to 65 years 
of age. “Time since last enrolled in school” could 


have been within the current year to as much as 50 
years ago. The first question is going to make some 
respondents have to stop and think. The objectives 
of the question might also have been served by only 
asking year of last enrciiment. 


In the second question, what constitutes ‘work’? Are 
we looking at any work or full-time work? Must the 6 
months be consecutive, or any 6 out of 127 


Although you may really have a desire to collect a 
certain piece of information. the accuracy and ease 
with which i can be reported to you must be 


TAG:: p. 50. 
ee 


Question A1, ‘Did you do any work for pay during 
that week?" sets up a branching sequence. People 
who say “yes” wil then be asked questions A2 
through B5. People who say “no" wil skip to another 
branch question Cé. 


Al 


That is a very simple situation. if you have an 
expanded questionnaire, your branching may be 
more complex. 


For example, let's say that, eer the USDOL 
mandatories, you have a sequence of questions 
which finds out more information such as availability 
of benefits from the current or most recent employer. 
These questions are only asked of persons e.nployed 
any time within the 13 weeks. You might end the 
survey with a series of questions on public 
assistance, which would be asked of everyone. 


The interviewer arrives at the public assistance 
questions from two different places in the survey. if 
the person was not employed at all in the 13 weeks, 
the next question will be the public assistance series. 
Wf the person was employed, the public assistance 
series will foliow benefits questions. 


When you write your questions, the transition from 
each point to the next question must be smooth and 
logical. in designing your questionnaire, always read 
R aloud, following all the diflerert paths that your 
respondents might traverse. 
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rn 
| Public Assistance 


' 
(end) 


Your questionnaire, # properly designed, should flow 
@s smoothly as a Mozart concerto. A good 
questionnaire is not just a list of questions. Thore 
should be a musicike flow from section to section 
and question to question, the art that you bring to the 
science. 


There are two kinds of time in the world. One is clock 
time. The second is perceptual tine When 
subjected to a badly written questionnaire, the 
respondent watches the clock, and that time moves 
slowy. Your five minute questionnaire feels like 
fiteen because time expands. It's like being on hold. 


In a good questionnaire, the respondent is spending 
his or her time listening to you. Your five minute 
questionnaire feels like two because time contracts. 
That is why good designers can successfully keep 
people on the telephone for 20 minutes or longer -— Rt 
does not feel that long. 


There are two easy things that can be done to 


prevet poor questionnaire design. One of these is 
to allow yoursell plenty of time to do something else 
and come back to the questionnaire later. This 
means, write the survey, pass & around, improve &, 
pass i around again, and then put ft away for a few 
days. When you return to your last draft, you should 
find that the “down time" provides you with a new 
perspective you might not have otherwise. 


The final check on your design is the pretest. 
Pretesting with at least 20 real respondents is always 


an exceliert idea because & subjects your design to 
real world conditions. 


Asking the pretest participants # there is anything 
they warted to say bul were not asked is an excellent 
technique to help you look at the survey through your 
responderts’ poirs of view. 


Writing a good questionnaire is noi simple, and 
writting a good, short questionnaire is far more 
complex than many realize. These principles of 
design apply to any questionnaire, not just for JTPA. 


Any survey is an imposition on someone else's time, 
time that has value to them. Whether your survey is 2 
miruges or 20, the respondent should finish feeling 
that 2 was worthwhile to trade their time to answer 


your questions. 3 a 
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To order a of the 1990 TAG., or to additional copies of TA.G.*, contact Steve Aaronson, 
Empiomart ard Traiing histor, US Department ot Labor Room N5310, 200 Constitution Avenue, 


Mike Dail of South Carolina has been gathering and a document of ‘the best quesiions® used in 
~_ae evaluation. Cortact him for 
1550 Gadsden Street; Box 995: Columbia, SC 


A useful document used in the federal government is Developing and . Program 
Division, Unded Stanee General Aco Acaourting Ciice. P.O. Box ef Box 6015, 


Evaluaiion and 
Gatthersburg, MO 20877. GAD PEMD 10.1.7.; (200) 278 204% 


There are many difierert books on is one of the best. Dr. Diiman's methods are widely 
ned, especialy se GAD trol enadios Dimer C Don A. Mail and Telephone Surveys: The Tota/ 


Design Method. John Wiey & Sons, 1978. 


Members of the American Association for Public Opinion Research may be in your area. if yu need a 
“second opinion” or want an outside expsrt to assist in design or analysis, it is worth making a call to ask 
them for a local resource -- (313) 764-1555. 


& You have a choice. When you do your foliow-up surveys you can sample your population 
if & is large enough, or you can interview everybody. Why or why not? 


| Did you know that the physical layout of your telephone questionnaire can increase or 
decrease accuracy of data? 


a Do you need to use a mall survey? What should you know about its design? 


a Finding resources to make the most of postprogram evaiuation data is a challenge. Have 
you tried these ideas? 


T.A.G. + 


Supplement to the Technical Assistance Guide for JTPA Follow-up and Validation - MOOULE 2 - 1962 


Published by ths Employment anc Training Administration, US. Ospartment of Labor, Fen 5310, 209 Constitution Avenues, N.W., 
Washington, 0.C. 20210 


Who you interview 'mpacts what you can do with your information, so we address the question of whether you 
should sample or interview the universe. Follow-up design issues are continued by discussing *»». \. -ormat a 
Questionnaire. 


We aiso review techniques for designing a good mail questionnaire for JTPA follow-up. Finally sJdress the 
problem of finding resources to conduct sound follow-up and postprogram evaluation. 


TO SAMPLE OR NOT TO SAMPLE — 
THAT IS THE QUESTION 


The decision to sample or interview everyone is not 
simple. Starting with the basics of sampling for JTPA 
follow-up is a good point of departure. 


TAG: pp. 23-25, 28. 
ee 


USDOL requires that a minimum sample be 
scientifically selected for each SDA and SSA. Even 
though your actual terminations may deviate from the 
planned number of terminations, your sampling 
percentage is established based on your annual plan 
and is nof to be changed during the year. 


CONSISTENT RELIABILITY 


The sampling percentage has been chosen to give a 
consistent reliability to your information froin year to 


year and from SDA to SDA, regardiess of the size of 
the termination population. 


CALCULATING SAMPLE SIZE 


The 1990 T.AG. (pg. 25) provides the chart to use in 
determining either the appropriate sample size or the 
sampling percentage to be used. 


There is no perfect way to control the sampling. 
Economic conditions and other factors do change in 


a local area over the course of a year. Your plan may 
meet, exceed, or fall short of correctly estimating the 
number of planned terminations. 


TAG.: pp. 28-29. 


No changes are to be made in your sampling ratios. 
This ensures that your information at the end of the 


year is representative of one year’s worth of activity, 


os TAG.: pp. 


Information in the 1990 TAG. is keyed” by this symbol. 


aa 


information in TAG * Modules le lagged” by this symbol 


Prepered for the U. S. Department of Labor, ETA by Appropriate Solutions, inc, Columbus, Ohio. Contract @99- 1-3654-72-078-01 


q 
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even if you miss your numbers. If you were to adjust 
the sampling percentage during the year there is a 
very real possibility of unintentionally biasing your 
sample. Seasonal variations do occur and 
constantly readjusting your percentages means you 
will miss this. in order to counter this potenmal 
problem we recommend that you increase the 
minimum sample size by five percent. 


CAUTION. You can sample using the 
sampling percentage in the chart, or you 
can establish your own percentage fy 
dividing the minimum sample size by the 
numbers of planned _ terminations. 
Calculating your sampling percentage 
will guarantee your sample will be large 
enough - # your actual number of 
terminations is close to your plan. 


The sampling procedure is designed to give you a 
margin of error of pius or minus five percentage 
points. USDOL wants to be certain that regardless of 
any error percentage you do have at the end of the 
year, your data properly reflects seasonal variation or 
unexpected economic “bubbles.” 


Sampling provides the lowest cost approach to 
follow-up. You have other choices, however. 


INTERVIEWING THE JNIVERSE 


At the other end of the continuum of choice, you can 
interview everyone. Why would you want to do that? 


T.AG.: pp. 26-27. 
ee ee 


if you are using your follow-up data to assess the 
performance of vendors, sampled data may not be 
sufficient. As the size of the SDA increases, the 
sampling percentage decreases. An SDA with 2,000 
terminations would sample only 352, while an SDA 
with 200 terminations would sample 175. 


Only 17% of the larger SDA’s vendors’ terminees 
would be interviewed, whe 82% of the smaller SDA’s 
terminees would be called. Vendor evaluation would 
then be much more accurate in the smaller SDA than 


in the larger. 


it is quite permissible to hold your vendors 
accountable for the performance of their graduates. 
By interviewing all terminees you eliminate 
discrepancies and allow a follow-up based vendor 
assessment to proceed with more confidence. 


For example, do not aitempt to base 
evaluation of a vendor on follow-up 
interviews concucted with only 2 out of 50 


of their participants. 


Another reason to interview all terminees is because 
you want to look closely at subgroups of your 
participants and need greater accuracy of your data 
at the subgroup level. Module 8 will discuss error 
and reliability in depth. 


OTHER CHOICES FOR SAMPLING 


You do have more choices, though, than just 
meeting minimum sampling requirements or 
interviewing everyone. 


USDOL specifies minimum sample sizes. if you want 
to go beyond the minimum, you should have a 
utilization plan. What do you want to do with the data 
beyond meeting Federal requirements? 


Here are some ideas. 


B You decide to survey all of the participants 
from vendors providing services for the first 
time and you use minimum sampling 
requirements for all others. You end up with 
more reliable information to determine the 
success of a new vendor's program. 


a You are running a new and innovative 
program. All terminees from this program 
will be in the follow-up so that the program's 
activities can be better assessed. All others 
follow minimum sampling rules. 


= As a larger SDA you want to get more 
accurate Information for both vendor and 
subgroup analysis, but you do not want the 
expense of interviewing everyone. Simply 
increase your sampling percentage. 
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Each of three examples results in better information, 
but at a higher cost. Analysis of your SDA’s data 
becomes more cornplicated, but you get exactly the 
kind of planning data thai you need. 


CAUTION: When you begin changing 
your sampling, how you assemble your 
data for reporting may change. Check 
with USDOL if you have questions. State 
level data will have to be weighted. 


TAG.: pp. 23-31. 


Everyone has to meet the minimum sampling 
requirements. What you do beyond that will depend 
on vour individual and specific needs. Your plan on 
how you wish to uss the Information must be 
specific. Your cptions are limited by your creativity, 
imagination ard resources. Follow-up, if limited to 
only filling out Federal forms, is an expense. You can 
tailor # to be a valuable addition to your planning and 
assessment activities. 3 sk 


[ FORM FOLLOWS FUNCTION 


In Module 1, we talked about how to write good 
questions for your postprogram surveys. The 
second part of getting good information back from 
the respondents is to make certain that your 
interviewers can conduct their interviews as easily 


and accuraiely as possible. 


Visual design and flow is very important. Visual 
design includes size of the type, amount of white 
space, line spacing, positioning of sets of questions 
en a page and visual isolation of instructions to 
interviewers. Do not underestimate the importance 
of your physical format in allowing your interviewers 
to collect good data. 


Perhaps a case study will better make this point. 


A cliem was conducting a survey and wanted a 
consulta to provide quality control and data 
analysis services. 


The cllem decided that, since temporaries 
would be used to conduct the interviews, they 
would do a better job if they didn't have to turn 


any pages. 


So the client put what should have been a 15 


to 20 page survey onto one 11x17 piece of 
paper. Classified ads would have been easier 


to read. 


To make matters worse, the client chose a 
bright fuchsia paper which gave readers 
headaches just looking at it, no less trying to 
read something on tt. 


Suffice & to say, the client’s entire analysis 
budget (and then some) was consumed in 
Cleaning the data -— that is, removing all of the 
errors mavie by the interviewers and by the 
keypunchers. Even then, the data weren't 
particularly accurate, but the client certainly 
did save a lot of money on his printing. 


Keep in mind that doing an interview is a stressful 
situation. The interviewer is supposed to be 
concentrating on whai he or she is asking and on 
what the respondent is saying. Straining to read the 
questionnaire, concentrating on accurately recording 
the response, and figuring out where to go next 
restrict the interviewer from the primary job. 


Improve the accuracy of your data collection by 
looking closely at how easily your questionnaire can 
be used. 


|| 
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The differences between the following two examples are stark. In the good example, instructions to interviewers 
are in bold. The numbered responses that an interviewer circles are offset to prevent accidental circling of two 


numbers. The questions and responses are not jammed together on the paper. 


BAD EXAMPLE 
The BAD EXAMPLE is reproduced in a size slightly 4 tS tee jes pie te 
larger than what the interviewer was provided! 20. team clhas enadio ave ts aur tamaaheaté? 
“ 47 1 Total 
x Over 15 
BP adtee Nags aw so 
= GOOD EXAMPLE 
The GOOD EXAMPLE shows the identical questions ™ area 
in a format and size much more conducive to ° -F 
accurate data collection and recording. %* 
17. What was your age on your last 
birthday? 
RECORD ACTUAL AGE 
968 = 98 or older 
| |__| 99 = Refused 


18. How many people are in your 


18a. How many of these people are under 
eighteen? 


|} 
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The T.AG. offers three methods for collecting the 
follow-up data--in person, over the telephone, and 
through the mail. Telephor.e and personal surveys 
are required except in those cases in which the 
terminege does not have a phone or cannot be 
reached. 


a T.AG.: pp. 49, 52; App. A, p. 15. 


In telephone and personal surveys, a trained 
interviewer is present to ask the questicns, provide 
explanations and probe for clarification as necessary. 
Personal interviewing is not used with any frequency 
due to its high costs although it has been used with 
some degree of success with rural populations that 
do not have ready access to telephones or are 
geographically isolated. 


No matter how good your telephone questionnaire 
response rates are, the use of a mail questionnaire 
can be valuable. Some people absolutely will not talk 
to you on the phone, while some of your people may 
also be incredibly difficult to locate for a telephone 
interview. 


T.AG.: pp. 52, 58. 
eee 


In one State, a mail questionnaire is sent out with a 
third mail contact in the third week of the five week 
window. Responses to this survey constitute around 
3% of this State’s completed interviews. These 
completions, however, are disproportionately from 
harder to locate populations. Although a small 
percentage, these interviews are an important part in 
reaching the minimum response rate. 


In another State, where the population is spread out 
over very large geographically disparate areas, a mail 
questionnaire is sent out with the pre-survey letter. 
While this practice would be heavily discouraged 
under other circumstances, this State is faced with 


the jollowing challenge. 


[ THE QUESTIONNAIRE’S IN THE MAIL 


A large percentage of terminees give post office 
boxes or general delivery as a home address. 
Many villages have but a single phone which is 
only answered during certain hours of the day. 
Mail delivery in parts of the State is difficult. 
Many terminees are on ships or in the 
wilderness for weeks at a time. By including 
the mail questionnaire in the pre-survey /etter, 
the State increases the chances of the terminee 
getting in touch with the research contractor. 


In terms of the accuracy of your information, you are 
always better to leave the collection of the data to an 
interviewer. Few things are more painful during 
follow-up than receiving a mail questionnaire and not 
being able to use & oecause the terminee did not 
answer a mandatory question properly. 


Designing a mail questionnaire is not simple and 
follows somewhat different rules from designing a 


telephone questionnaire. 


A Good PHONE SURVEY IS NOT A GOOD MAIL SURVEY 


First, do not simply photocopy one of your telephone 
questionnaires and send it to a terminee. If you did a 


good job designing your telephone questionnaire, tt 
is going to have instructions to interviewers and other 
things that make it a good telephone questionnaire. 
Those things don't belong in a mail questionnaire. 
There should also be specific instructions for the 
person filling out the mail questionnaire. 


NEAT, LOGICAL, UNAMBIGUOUS 


Second, the questionnaire must be unambiguous. 
No interviewer is present to answer questions. 
Everything must be neat and totally logical. The 
questionnaire must be easy to understand for those 
with lower reading skill levels, while not insulting to 
those with higher skis. 


INCLUDE PROBING QUESTIONS 


Third, you shouid ask other questions that you rely 
upon your interviewers to capture with probing. For 


Is 
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instance, you need to make some provision for 
recording extra earned income. For example, 
overtime information is helpful since you might need 
that to calculate weekly earnings, or the adjusted 
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questionnaire is longer, the mail questionnaire is your 
last resort. Carefully consider what information, in 
addition to mandatory questions, you feel is very 
important and which your program participants can 


hourly rate for EDWAA. answer easily and accurately. 

No MORE THAN Two PaGeEs Pay For POSTAGE 

Fourth, do not exceed two pages in length (or one Fifth, always include a postage paid return envelope 
page double sided). Even if your phone for convenience. 


= = 
TiS EXAMPLE SHOWS A MAM QUESTIONNAIRE THAT HAS BEEN GENERATED ON A LASER PRINTER WHICH MERGES I THE APPROPRUTE DATES. This 
CAA BE RETURNED IN A POSTAGE PAID ENVELOPE OR DESIGNED AS A SELF MAILER. 


If THE REVERSE PAGE 18 AVAILABLE FOR SURVEY QUESTIONS, YOU CAN INCLUDE ADOITIOWAL QUESTIONS FROM YOUR PHONE SURVEY. KEEP IT 
SHORT THOUGH, AND NO LONGER THAN TWO PAGES. 


JTPA/PIC EVALUATION 


socee Hosen Soe Quoetion Canc be Aen Fats Mile Se Pe. he Sereay Cv Detertens, Pence Cut POenpanyl Meee. in 


1. The first few questions deal with employment. We are looking at a 13 week period between [dates]. Were you working at ail for 
pay during any of those 13 weeks? This would include any weeks in which you were on paid vacation or sick leave. Check 
Yes or No. 


{ ] YES { ] NO - Please go to question 7 at the bottom of this page. 

2 How many of those 13 weeks did you work? 
| worked weeks (1 to 13). 

3. Were you employed during the week of Check Yes or No. 
{ } Yes [ ] NO - Please go to question 7 at the bottom of tiils page 

a How many hours did you work for pay during the week of [Reference week dates]? Fill in the blank. 
| worked hours. 

f you worked more than 40 hours, were you paid extra for the time worked over 40 hours? Check the box that applies. 

[ ] Time and a half [ ] Double time [ ] Other: 


if you were not paid per hour, check this box [ }. 
5. How much were you paid per hour in that one week? Fill in the blank. 
twaspeid = =—s_—($s —S§$—Ssérpor hour. 


6. Altogether, how much did earn for work that you did in that one week before deductions? This would include money 
camed tom el jobs and work you may have dane on your own. Fill in the blank. ™ 


| earned doliars in that week. 
7. The next few questions deal with financial support have received for the month of Were 
receiving any of the following: (Please circle your snewer} _ a = 
YES Monthly Amount 
a. Aid for Dependent 
Children (ADC)? 1 2 $ 
b. General Retief? 1 2 $ 
c. Food Stamps? 1 2 $ 
d. SSI Benefits? 1 2 $ 
e. Medical Card through Welfare? 1 2 
t. Subsidized Housing Assistance? 1 2 


EXAMPLE: NOT TO SCALE 


4 


—————————————— 
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TIMING IS IMPORTANT 


Sixth, the questionnaire should be mailed out no later 
than the end of the third week of the window in order 
to allow time for 8 to be completed and returned to 
you in time. 


INCLUDE HELP NUMBERS 


Seventh, both the cover letter and the questionnaire 
should contain a “help” number the person can call 
for assistance. 


If you get them on the phone, use the phone survey. 
While the mail survey itself may only generate 3% of 
your completions, & will easily generate another 3% 
by stimulating call-ins. 


The mail questionnaire can be a valuable addition to 
your research program. Design kt carefully. Always 
look at it from the perspective of the person who has 
to fil t out. Anticipate possible questions the 
respondent will have and make sure your design 
accommodates them. 3 


There is no question about it. A properly designed 
and conducted follow-up process is expensive. You 
spend a lot of money just to acquire the numbers 
which go into the JASR and WAPR and into 
performance standards calculations. 


The added cost to move from “expense” (collecting 
the data) to “investment” (using the information) is 
modest. Yet in many States and SDAs, there are 
barriers to this movement. 


One of those barriers is lack of utilization experience 
and models to follow. 


A key component in improving utilization ts 
expanding your networking and sharing experiences. 
Talk to people in other SDAs and States who are 


beginning to actively use their database. 


FIVE WEEK WINDOW?? 


Not everyone is aware of the USDOL rules 
conceming the interviewing windows. Read item 
2 on page 33 of the 1990 TAG. 

You have 4 weeks to locate your terminees. 


VOC. 4°. ° RL th as LECH F Wet tt 142 OL4-4 
the interview. if you use a ‘last resort’ mail 
survey, the fifth week is very important to you. 
You can increase your response rates 2% to 10% 
by using the fifth week. 


What is ‘located’? If your mail contacts are not 
returned, it is reasonable to assume your address 
is correct unless you have been informed by 
someone that the terminee has moved — or — 


if you have a valid contact number, you have 
located the terminee — or - 

if you are in direct contact with an individual who 
is in contact with the terminee, you can consider 
the terminee to be located, even though you are 
at this point not specifically talking to the 


SK FINDING RESOURCES 
" 


J] an 


Many national, regional and State conferences are 
available. If you are in a State with a training institute, 
special courses may be available to you. 


Look for organizations which you can join that link 
you to others with similar concems. The 
RESOURCES section will give you a couple of ideas. 
There are models and examples out there, but you 
will need to look for them. 


A second barrier to utilization is how to pay for t. As 
you increase the amount of useful information you 
get from your database, you will find the cost/value 
ratio improving. This does assume you have the 
ability to manipulate and analyze your data. 


But even with limited funding for computers, 
software, and talent, you still have options. 


IS 
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Look toward your PiCs. PICs not only have 
resources within thelr own organizations, they likely 
have many connections within the community. 


PICs are important consumers of follow-up 
information, and key partners in the JTPA system. 
The more they know about what you do, the greater 
will be their interest in what they can learn and use. 


Donated services are also a possibility. Look to 
colleges and universities as sources of expertise, 
especially your community college system. in the 
community colleges, instructors often teach part time 
while also having active careers in the business 
community. Their blending of academe and business 
may provide them a very pragmatic perspective. 


Having the opportunity to take on projects “in the real 
world” is often very attractive to the professors for 
their own research as well as for practical experience 
for their students. Many stil contribute thelr time 


willingly. 


Local foundations may be interested in funding 
special projects. 


The fact of the matter is, just because your budget 
today does not support what you would like to do, by 
creatively exploring options, you might be surprised 
at how much support is really out there. 3 


Erdos, Paul L, Professional Mail Surveys. McGraw-Hill Book Company. Many good ideas. 
The Partnership for Training and Employment Careers; 1620 Eye St., N.W., Washington, D.C. 20006; (202) 387-6120, 


-- $45/yr. This professional 


Directory of Statewide JTPA Associations, Michigan 


has numerous useful services. 
Works, Michigan Job Training Partnership Association; 2163 


Jolly Road, Suite 101B, Okemos, Mi 48864; (517) 349-6510. Somewhat dated, but excellent networking source. 


Lagden ph penmmennng lene n | tfc arene Ape pm nen man anydin ye plan nm One interest group 


is composed of persons interested in 
Evaluation Association; Job Training 


in Social aeons and Industry Topical 


and their assessment. American 
yea oe Menno 


Solutions, inc.; 1357 W. Lane Ave. - Suite 207; Columbus, OH 43221; (614) 486-5295, - 


Improve your location process with a well designed terminee call record. 

Finding terminees is a major cost of follow-up. Are you maxing it easy for them to find you? 
Time is money. Your survey infrastructure can save you both. 

What are operational statistics and what should you be doing with them? 

What are the pluses and minuses of paying respondents? 


T.A.G. + 
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A well designed record for tracking location efforts pays off in lower costs. We provide some ideas on organizing 
your follow-up process. By studying operational statistics, you can refine your program, reduce costs, and improve 
efficiency. What are the advantages, and the pitfalis of paying terminees for follow-up interviews? 


| TERMINEE CALL RECORDS | 


Having a separate call record for each program 
terminee is a good idea. A well designed call record 
provides the interviewer with as much pertinent 
information regarding the terminee as possible. It is 
also the collection point for operational statistics and 
a master log of all actions taken to locate the person. 


TAG.: p. 47. 
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There are many advantages to having a call record 
for each individual instead of working off of a 
terminee list or incorporating elements of the call 
record into the questionnaire. The call record should 
be a full single sheet of paper with at least ten lines 
for notes and contact information. Note taking is one 
of the most important things an interviewer does. 


Did you talk to a mother, brother or cousin? Whose 
number did you call? What did you find out from 
your call? Do you need to call back at a different 
time? Is there a new number at which you can reach 
the terminee? Has the terminee moved? i's the 
person still employed at the same company as 
records indicate? 


The notes section provides you with the building 
blocks for deci)’ what your next steps will be. The 
importance of ifs becomes apparent when you are 
the next person that needs to make a call for a 
terminee and the notes are so vague that you have to 
start at the beginning and go over old ground by 
calling an unproductive telephone number. This is 
imacrtant even if only one interviewer is making the 
calls. Don't assume you will remember everything 
next week. 


The terminee call record provides the interviewer with 
a portrait of the person that they are trying to locate 
and interview. It is a centralized point of information 
that is continually updated by the interviewer after 
each contact attempt with the terminee. All important 
details of contacting the terminee should be noted. 


Interviewers require a base of contact information 
and participant information in order to perform follow- 
up effectively. The following is a general list of what 
should be on a terminee call record at a minimum. 


fe T.AG.: pp. 


information in the 1990 T.AG. is "keyed" by this symbol. 


“= 


information in T.A.G.* Modules is “flagged” by this symbol. 


Prepared for the U. S. Department of Labor, ETA by Appropriate Solutions, inc., Columbus, Ohio. Contract #99- 1-3654-72-078-01 
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month and year of birth/age 
limited English 


terminee telephone number - updated at 
intake and termination 

message number - primary - updated at 
intake and termination 

message number - secondary - different 
from primary - updated at intake and 
termination 

employer name 

employer telephone number 

title of program enroliment 

program activity - primary 

type of termination - positive, negative 

date of termination 

reference week 


The minimum list is very basic and yet it provides 
information that is helpful in successfully contacting 
and interviewing the terminee. If we were to expand 
this list of call record items we might include: 


Most follow-up surveys are conducted by telephone, 
the majority through outgoing calls. Incoming calls, 
however, are extremely valuable. if you have no 
capacity for taking incoming calls you will find it 
difficult to reach the required 70% minimum response 
rate. 


As JTPA increasingly provides training to hard-to- 
serve populations, the percentage of persons with no 
home phone numbers, as well as no messa~* 
numbers, will increase. When the telephone numbers 
provided to the interviewers do not easily result in 
completed surveys, the pursult process gets more 
expensive because other location techniques have to 


message number - primary 
contact acd relationship 

message number - secondary - name of 
contact and relationship 

vendor telephone number 

vendor contact 

employer contact person 

employer address 

aciivity invoivement - list 

support services - list 

area for coding the accuracy of terminee 
and message numbers 

area for coding the total number of 
contacts and final disposition 

area for coding the effectiveness of 
alternate location strategies. 


- name of 


When you are dealing with larger follow-up 
populations and have more than one interviewer on 
Staff, the terminee call record takes on even greater 
importance. Any interviewer should be able to pick 
up a call record and easily follow the actions that 
preceded their picking it up. This requires good note 
taking on the part of all interviewers. Notes should 
indicate and highlight important aspects of when and 
how to contact the terminee. You will appreciate this 
system when a terminee calls you. %* 


be used. The greatest percentage of your cost will 
be due to pursuit, not to the actual interviewing 


Through a combination of contacts by mail and 
leaving messages for the terminee, ya; should be 
encouraging that person to call you. It is important 
to have a system to efficiently handle these calls. 


THE LONG DISTANCE RESPONDENT 


Since follow-up was started in 1986, many sites have 
discovered that the terminees did not want the 
inconvenience of dealing with an operator. Adding 
800 numbers solved the problem by making it free 
and simple to call the researcher. 


If 


TAG.: pp. 44-45. 
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Up until the mid-1980s, 800 numbers were very 
expensive and required a dedicated telephone line. 
You needed to be taking hundreds of calls to make 
them cost effective. 


Today, you can get an 800 number which comes in 
on your regular phone lines. if you have multiple 
phone lines which rotor (that is, a call coming in on 
your main number will skip to the next available line if 
the main number is busy}, each line is capable of 
taking an 300 call. 


An 800 number can even be used at the SDA level, 
depending upon the geographical size of the SDA 
and number of persons served. One State reporied 
that call-ins more than doubled when an 800 number 
was put into service. Costs are extremely 
competitive. Check out your local options. 


Most sites should know the migration pattern of their 
terminees. if the vast majority of your terminees 
move in a certain direction, make sure your phone 
system covers that area. Those terminees outside 
the 800 calling area will have to call collect. 


The terminee who calls you to complete the interview 
will probably have different characteristics and 
experiences from those you reach on a call-out. 
Having a system to accept calls ensures the 
representativeness and accuracy of your data. For 
instance, one State reported that weekly income for 
call-ins was 4% higher than for call-outs. 


things due yesterday, answer the phone. 
This may be the one and only chance you 
will have to interview that person. 


| If you cannot fing his or her records, the 
terminee may be using a different name now, 
may already have completed a survey, or 
may not yet be eligible to be interviewed. 


See the next section for more hints on how to make 
your internal operations efficient. 9 3 


[ ORGANIZING FOR INTERVIEWING - 
BASIC INGREDIENTS | 


———-" 


ns 


One of the principles in maintaining an organizational 
process is that its failures will be more evident than 
its successes. When the system works, it is invisible. 
This section provides some hints and ideas on how 


you may be able to improve upon your existing 
operation to increase efficiency and lower costs. 


| 
0} 


T.AG.: pp. 17-19. 
Te 


The main difference between SDA-collected data and 
State-collected data is volume. The higher the 
volume, the greater the economy of scale for some of 
our ideas. Tracking 1,000 people a week requires 


Iq 
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more formal organization than does tracking 20. 
Adapt these suggestions to your particular situation. 


Your center is set up to meet the needs of two 
different people: 1) the terminee and 2) the 
interviewer. Look at ali procedures and processes 
from each of these perspectives. 


Hours OF OPERATION 


The greater the number of hours whhin which an 
interview can be completed, the higher the likelihood 
that you can reach an individual or be reached by 
him or her. 


The days of the week that you are available will also 
make a difference. You could be available 8 hours a 
day, Monday through Friday, but stili miss a segment 
of the follow-up population. 


Therefore, it is important that some evening and/or 
weekend times be available. There are several 
reasons for this. 


First of all, when you are calling employed terminees, 
day calls have little value, uniess you know they are 
on second or third shift. Do not waste precious time. 


Second, long distance calls will be less expensive 
during evenings and weekends. 


Third, one third or more of your surveys can be 
completed because the terminee called you. Make 
the call-in hours as convenient for the terminee as 


possible. 


While individual SDAs can make evening and 
weekend calls, being consistently available for call- 
ins during these times can be burdensome. 


MASTER LIST 


The larger your operation, the mor: ‘portant it is to 
keep on top of whom you are trying to reach. One 
way is to keep a master list containing the names of 
everyone within your interviewing window, as well as 
those to whom you have mailed pre-survey letters, 
but whose interviewing window has not yet opened. 


Each week, remove the names of those participants 
who have completed an interview o7 whose windows 
have closed. A simple list is alphabetical by the 
terminee’s last name and includes the reference 
period and an identification number. When a call 
comes in, this list can point you to where a caller's 
record is kept. 


TERMINEE CALL RECOROS 


if you have a small operation, you can keep all active 
call records stored alphabetically in one location. 
State level operations, however, may find that it is 
more efficient to keep call records clustered by 
reference weeks, with a separate file or set of files for 
each open reference week. 


One idea is to run your call records on five different 
colors corresponding to the five weeks of the 
interviewing window. This helps keep the weeks 
visually separate. The records can be alphabetized 
and kept in file folders or alphabetic desk files which 
are easily accessible to the interviewer. 


WHAT IS A REFERENCE WEEK? 


USDOL permits two different i 

windows beginning in the 14th week after 
termination. The primary window is weeks 
14 through 17 -- a four week period. A fifth 
week - #18 - is permitted in those cases in 
which the terminee has been located but not 
yet successfully interviewed. 


PAPER MANAGEMENT 


The larger the operation, the more cumbersome 
becomes the task of managing the paper. But even 
in small follow-up systems, misplacing somebody's 
records poses serious problems. 


ideally you will arrange your interviewing operations 
and materials in clusters. This is most important 
when you take a call from a terminee and have to find 
his or her materials quickly. 


The simplest organization is to keep your call records 
in four clusters. The active terminee call records, 
ones that you are still making outgoing calls on, 
should be storod together. This cluster includes 


ge 
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those cases in which you have recently lei a 
message and are hoping for a return call. 


T.AG.: pp. 33-35. 
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A second cluster is for those call records for which ail 
telephone pursult options have been exhausted. 
These people might sill get an old message or read 
the letter you sent and call you. 


A third cluster contains those for which special calls 
are to be made (Le.. Wednesday at 4 PM). 


A fourth cluster includes those for which you need to 
do some other pursult - look up phone numbers of 
possible relatives, call an information operator, check 
back with the JTPA office, call a welfare case worker 
and so on. 


The system can be as simple or detailed as meets 
your particular needs. The important thing to 
accomplish is to make sure nothing is misplaced and 
that the right materials can be found within seconds 
of receiving a respondent's call. Your respordents 
will appreciate your efficiency, and it may increase 
your chances of a completion. 9 3 


» OPERATIONAL STATISTICS - a 
HOW YOU DO WHAT YOU DO 


In racing, the two primary performance measures are 
average speed, and finishirx;. In some ways these 
are equivalent to JTPA performance measures. 


Along the way, however, a good race crew is 
monitoring literally hundreds of additional measures - 
- how long the tires last, speed of the pit crew, fuel 
consumption, etc. Assessing these measures goes 
into fine-tuning the machine and the system in order 
to maximize the primary performance measures. 


The JTPA follow-up system can collect operational 
statistics (efficiency measures of your follow-up 
system) that have the equivalent benefit. Since 
follow-up and postprogram evaluation are expensive, 
improving the efficiency of your survey operation can 
yield direct financial benefit to any site in addition to 
improving the accuracy of the data collected. Many 
sites record operational data, but have not put them 
into the computer with the survey data, and have 
never looked to see what they say. 


Operational data is collected from two documents -- 
the terminee call record and the questionnaire. The 
following are examples of information you can 
collect. 


From the Call Record: 


Terminee Phone number 

(none, wrong, disconnected, OK) 
Message or Collateral number 

(none, wrong, disconnected, OK) 
Employer number 

(none, wrong, disconnected, OK) 
Number of contact attempts 
Number of messages left 
Pursuit by phone book 

(not used, used helpful, used not helpful) 
Pursuit by criss cross directory 

(not used, used helpful, used not helpful) 
Pursuit by information operator 

(not used, used helpful, used not helpful) 


o { 
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interviewer ID number 

Coder ID number 

Supervisor ID number 

Source of completion (call out, call in, mail) 
Day of week 

Time of Gay 


You can merge this information with the survey data 
and with the MIS data creating two files -- one for all 
sampled terminees, and one for completed surveys. 


The information can now be analyzed by SDA, by 
demographic groups, and by status at termination, 
just to mention a few options. The following are 
some ideas. 


TAG.: pp. 45-46. 
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NUMBER OF CONTACTS 


First of all, the 1990 T.AG. indicates that at least 6 
contact attempts should be made. if one State had 


stopped at 6 attempts, response rates would have 
only been in the 60% range. 


This State originally did not cap the number of 
atternpts. Anything in the five week window was 
fair game. Analyzing their data, however, clearly 
showed where the point of diminishing returns 
occurred, and with what demographic groups 
and in what SDAs. The location process was 
modified to put a general cap on pursult at 15 
atterm vs with several exceptions These 
exceptions are related to particular 
demographic groups and SDAs in which more 
aggressive location activity was necessary. 


The 1990 TAG. (pg 37) says that the “same 
effort should be made to contact all types of 
terminees regardless of their termination status 
or demographic characteristics." However, 
experience has now shown that consistent 
accuracy of data across different groups will 
require some special effort. 
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For this reason, 
recommendation 


changes _ that 


For your data to have Comparable accuracy. 
Cifterential pursuit activities may be 
necessary. One size does not fit all. Study 


Second, there is a direct correlation between the 
number of telephone points of contact that are 
provided to the interviewers and the final response 
rete. One State reports an average response rate of 
89% when they start with threz potential numbers, 
dropping to 43% when they are given no numbers. 


COST OF LOCATION 


Third. the cost of location goes down as the number 
of telephone numbers initially provided goes up. 


Looking to the number of contact attempts and 
the use of additional pursult strategies enables 
you to determine if certain SDAs or subgroups 
are causing expensive pursuit situations by 
failing to provide telephone number and address 
information. 


In reality, having good telephone contact data 
on all of the participants is a goal that cannot be 
reached. However, most SDAs could do a 
better job at getting this informaticn at intake 
anc updating it at termination. 


TAG: pp. 38-40. 
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You may want to consider implementing a 
procedure in which the SDAs collect additional 
information on those persons who ff the 
difficult-to4ocate profile which your operational 
statistics provide. This additional information 
can be kept in the individual's fle and mad« 
accessible to the survey interviewers #f initial 
location fails. 
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Fourth. use of criss cross directories is more 
successful in some communities than in others. 
Differert sources of data are used to compile these 
directories. 


A phone book lists people alphabetically with 
their phone number. A criss cross directory lists 
people by street address and by phone 
numbers. ‘t is used to verify addresses, get 
names that phone numbers are listed in, and 
identify message numbers. 


Criss cross directories are produced by private 
companies, not your telephone company. 
Consequently, these directories may not be as 
timely or accurate as your phone book. They 
nevertheless do have benefit as long as you 
recognize their limitations. 


Criss-cross directories are expensive, costing 
hundreds to thousands of dollars per State. 
Operational statistics can show you what areas 
of your State or SDA would benefit most from 
using these as a resource. Lease or buy only 
those that are cost-effective. 


The rationale for paying terminees to complete a 
follc\/-up interview is that more people will respond, 
have seen no research which tests that assertion. If 
the participant is properly prepared for the follow-up 
survey, payment should be unnecessary. When 
money is introduced into research, potential new 
problems arise. 


Equity 


Eirst of all is equity. Are you offering the same 
payment to different categories of people or are you 
Offering more to members of hard-todocate 
categories? if terminees ever talk to each other, you 
could set yourself up for the easier-to-iocate people 


Cm | Cn 
oe PAYING RESPONDENTS os 


Check your Yellow Pages under “Publishers- 
Directory & Guide” to find out where your local 
books are available. 


Since locating terminees is the most expensive part 
of follow-up, understanding what is happening does 
have benefit to structuring a follow-up system that 
makes optimal use of scarce resources. if you are 
already recording this information, but not merging it 
with your cther data, with litiie effort you can improve 
your understanding of your operations. if you are not 
recording this information yet, doing so adds very 
little to what is already an involved process. Turn the 
cost into an investment. 


Examining the efficiency of your operation 
necessarily takes many forms. One of these is 


systematically collecting and analyzing operational 
statistics. 


A few sites have felt they improve their location and 


interviewing process by paying respondents. There 
are some cautions to this. 9% 9 


holding out for more cash. The same incentive 
amount should be offered to everyone. 


Second, are you offering payment at the beginning 
of the interviewing window or are you waiting until 
some number of unsuccessful contacts are 


attempted? Again, assume the word is out. 
the impact if terminees know they'll get paid 

if they ignore you for two weeks. 

IDENTIFICATION 


Third, are you sure the person answering the 
questions is the right person? When ‘ou add cash to 


a’ ” 
AD 
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the equation, you provide new and different 
incentives. 


if you are going to pay respondents, you need to 
establish a method of ensuring that you are talking to 
the right person and thai the proper person receives 
payment. Such a method should be based on 
something that the terminee will know but that not all 
others would easily know. Some locations use 
birthdate. Others use social security number. 


A combination is possible. Financial institutions for 
security purposes use am individual's mother’s 
maiden name. That data, however, is probably not in 
your MIS. Just be careful of the incentive you are 
Offering, and the opportunity this provides for 
someone other than the respondent to take 
advantage of you. 


CAUTION. Verification of identity when 
using a mail survey becomes a concern 
when money is involved. Use payment 
for mail surveys only if you have tested 
and validated your identification system. 


CONTROLS 


Fourth, what are your financial controls? Are people 
paid in cash or by check? Do you know that the 
respondent actually received the money? Do you 
have procedures to track any complaints about not 
receiving the money? 


You should assess your process and determine 
whether it helps or is injecting bias into your 
response rates. You should also document the total 
cost of this process and compare it with alternative 
methods to achieve similar response rates. Total 
cost includes the value of time and materials used in 
order to design, administer and monitor your system, 
not just the payment to the terminees. 


Since payment to respondents is done to enhance 
location and response rate, these costs are totally 
chargeable to your administrative budget and cannot 
be considered a supportive service payment. % 3 


Dillman, Den A. Mail and Telephone Surveys: The Total Design Method, Chapter 7. John Wiley & Sons, 


1978. 


Earl R. Babbie, Survey Research Methods. Wadsworth Publishing Co., 1973. A good, readable book on 


survey research. 


COMING IN 
MODULE 4 


a Are your participants prepared for the follow-up survey? 


a The greatest follow-up cost is locating the terminees. Helpful hints and ideas are shared. 


a When you call your terminees or possible relatives how do you identify yourself? 
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IN : 


THIS 
— MODULE 


Doing follow-up is a laborious process. This is made easier if your terminees know that they will be asked follow-up 
questions and why. Are you preparing your participants for follow-up? Being notified about the survey just before 
it occurs greatly improves your response rates. 


Location, not the interviewing, is the biggest challenge of follow-up. Techniques that you can use to your 
advantage in locating terminees are discussed. 


Finally, how do you identify yourself when you are trying to locate a terminee? Do you run the risk of violating any 
privacy restrictions? 


| PREPARING THE RESPONDENT FOR FOLLOW-UP | 


The SDA has the responsibility for collecting the 
information necessary to locate the participant after 
termination and to make certain that the participant 
understands why he or she will be contacted. 


Example: ‘The follow-up is very important. It lets us 
get your honest opinion on how good our services 
were and where we can make improvements. It also 
lets us collect the information that Congress wants in 
order to measure our effectiveness." 


T.AG.: pp. 37-40. 
rT 


The first notification about the follow-up survey 
should occur at intake. The applicant provides 


address, phone number, and one or more message 
numbers (see box on next page). 


At the time the person leaves the program, all contact 
information should be updated. The best clue to the 
people doing follow-up that no updates occurred at 
termination is when a letter is returned from the Post 
Office with the notation “Forwarding Expired,” -- 
meaning the address is over 18 months old. 


The 1990 T.A.G. shows one example of a follow-up 
agreement that can be given to people at intake. 
Although people are agreeing to participate, their 
participation is still totally voluntary. 


Some people may be members of a harder-to4ocate 
population and require additional contact 


EF T.AG.: pp. 


Information in the 1990 T.AG. is "keyed" by this symbol. 


a 


information in T.A.G.* Modules is “flagged” by this symbol. 


Prepared for the U. S. Department of Labor, ETA by Appropriate Solutions, inc., Columbus, Ohio. Contract #99-1-3654-72-078-01 
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information. The form taken from the 1990 T.AG. 
can be completed at intake and used at follow-up. 


This can be stored in the individual's file and be made 
available to the research crew if necessary. Such 
storage means that the MIS does not have to be 
adapted to hold the extra contacts. 


The third opportunity to prepare the respondent for 
follow-up is at the time the pre-survey letter is mailed. 
That is Advance Warning. 2 3 


WHAT IS A MESSAGE NUMBER? 


Practices differ widely among SDAs 

and States. At a minimum, one phone 
number in addition to a home number is 
collected. This should be a friend, neighbor 
or relative who would be able to get a 
message to the applicant or would know his 
or her whereabouts. This person does not 
need to be in the same city as the applicant. 


The most thorough systems collect more 
than one message number (also called a 
collateral contact) and may collect that 
person's name, address and relationship to 


the soplicant. 


| ADVANCE WARNING | 


lf there is a guru of telephone and mail survey 
research methodology today, one leading candidate 
would have to be Dr. Don Dillman at Washington 
State University. Dr. Dillman’s techniques (called the 
Total Design Method) have been widely adopted by 
the U.S. General Accounting Office to guide many of 
its research studies. 


The sequence of mail contacts with terminees, 
described in this module, is an adaptation of the Total 
Design Method (TDM). These are techniques that 
work. 


The first step, and one of the most important, is the 
pre-survey letter (PSL). These are the principles of 
TDM that should be accomplished within this letter: 


a Personalize your mailings. This is not junk 
mail. Emphasize the importance you put on 
talking to that individual. 

a Explain the value of the research. 


| Indicate the importance of this individual's 
contribution to the success of the survey. 


a Emphasize confidentiality. 


a Convey any other important messages. For 
example, state that participation is voluntary. 
You may even list some of the questions 
and/or verify location information. 


TA.G.: p. 42. 
ee 


In addition to improving your response rate, the PSL 
will also save your interviewers time. The terminee 


knows to expect the call and why the call is being 
made. The interviewer will have an easier job 
beginning the survey and will have fewer 
explanations to provide when the terminee says “Oh 
yeah, | got yc ur letter”. The PSL is also one of your 
best ways to assist in locating your terminees. 


Your 1990 T.A.G. provides one example of a PSL 
which could be used by an SDA doing their own data 
collection. The following is a sample letter you can 
use with an independen contractor. 3 3 


TO: [name of participant] 
FROM: [contact person] 
DATE: {date} 


weil they are performing. 


complete the follow-up interview. 


it to us. 


90000000, 


Council (PIC), 
unions, or the State Bureau of Employment Services (SBES). 


The U.S. Department of Labor requires every State to contact people who were involved with 
these programs in order to determine what impact the programs are having and measure how 


The State of [State name] has hired [contractor name] to do these brief follow-up interviews by 
telephone. [Contractor] is an independent research company which has no connection with the 
State of [State name] or its job training programs. 


We will be cailing you a‘ter [first date of interviewing window] to ask you a few questions about 
the services you received and your empioyment status during the past three months. Even if 
you were not working or did not finish a program it is important that we speak with you. If you 
found a job on your own we are still very interested in your opinions and it is still vital that we 


Since only a few people in each county are being calied, your opinions and answers do count a 
great deal and are very helpful. What we learn from you will help to improve the programs. 


We will be calling you at the phone numbers you gave the State of (State name]: 


Home: [home number] 
Message: [contact number] 


if either of these numbers is wrong or changed, piease fill out the enclosed postcard and return 


We will be calling you after [date specified). We cannot interview you before that day. if you do 
not have a phone and live in [State name} please call our toll free number 1-800-xxx-100x, 
Monday through Thursday between 9 A.M. and 9 P.M after [date specified]. If you live outside 
{State name] and the toll-free number does not work please call us COLLECT by dialing 0-xxx- 


Although your participation is voluntary, it is of great importance and all information is strictly 
confidential. Thank you for taking the time to share your opinions and helping us help [State 


LOCATING THE ELUSIVE 
RESPONDENT 


T.AG.: pp. 45-46. 
ed 


it should come as no surprise to anyone who has 
been involved with JTPA, that locating program 
terminees in order to conduct interviews can be very 
difficult. Statistics have consistently shown that 
refusal rates among located terminees are extremely 
low. This clearly indicates that the real work is in 
trying to locate and finally make a contact with these 
individuals. 


Many terminees will be located quickly and easily 
through very few attempts using the telephone 
numbers and address which they have provided for 
us. However, it is also a reality that many of the 
people involved with JTPA programs are highly 
mobile, and for many reasons may prove to be hard 
to reach. 


Telephone numbers may have been changed, 


disconnected, or possibly even recorded incorrectly 
during intake or data entry procedures. The terminee 


x! 
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may have moved, entered into military service, be in 
the hospital temporarily, or even be incarcerated. 
These are people just like anyone else, and the 


For these reasons, many respondents may appear to 
be elusive when actually they are unaware that you 
are trying to reach them. This is why it is very 
important to maintain a methodical, thorough, and 
persistent approach to tracking, locating, and finally 
contacting a terminee. 


There are two primary methods of location pursuit 
which are proven to be effective in achieving required 
response rates: telephone and mail. But keep in 
mind that “initiative’ will always be an important 
element in the detective work required to locate 
terminees. It is impossible to foresee all the 
situations you may encounter because each one will 
be different. Creativity and resourcefulness are 


encouraged in all cases. 


To begin the process, a Pre-Survey Letter should be 
mailed to the terminee’s address seven to ten days 
before the interviewing window opens. 


The PSL is individually personalized by name, 
identifies the sender and purpose, prepares the 
terminee for the interview and requests a response. 
Include a business reply postcard, so that the 


respondent can update any phone number or 
address changes. 


BEGIN TELEPHONE PURSUIT 
Begin making calls to the telephone numbers 
provided by the terminee. it is important to 


remember that direct contact over the phone is the 
most valuable and preferable form of completion. 


If all goes smoothly, the terminee will be reached and 
successfully interviewed by phone. When this is not 
the case, it wil! now be necessary to pursue and 
collect as riuch additional contact information on the 


responde’ it as possible. 


ABOUT TELEPHONE PURSUIT 


a Check phone books to find a new or correct 
number for the terminee. Try to find the 


terminee first. in best case scenarios the 
number will be listed.at the same address as 
the one you have in your records. 
Remember to look for initials when the name 
itself does not appear. You may find the 
phone number you have is off one digit or 
should be in a different area code. 


a Especially with female terminees, the 
female's first name may be listed as it 
appears on your call record, under first or 
first and middie initial, or with a spouse. For 
example, Alice B. Cannon could be found 
under: 


Cannon, Alice B 

Cannon, A. 

Cannon, A. B. 

Cannon, Ted & Alice 

Cannon, Ted (at respondent's address) 


Regardiess of first name, always scan for 
addresses, and then for phone numbers. 


When looking for a female terminee in the 
phone book, you may find the phone number 
at the correct address listed to her husband. 
Note this on your call record. A cail to an 
information operator would request a new 
listing for either wife or husband, now that 
you have both names. 


a Next, try to locate relatives of the terminee. 
These are often people with same or similar 
names, and also the same initials in 
particular. 


Note: A recent Gallup Poll reported that 67% of 
people with at least one living parent, live less 
than an hour's drive from Mom or Dad, and 54% 
see a parent at least once a week. 


| Also scan the addresses to find if anyone 
with the same last name as the terminee 
(possible relative) lives on the same street or 
road. There will also be times when the 
exact address will be found listed under a 
different person. This may be the head of a 
household or the person who happens to 
have the phone bill in their name. 


Ht 
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a Another method for finding a terminee’s 
relative is to find those people with the same 
last name and the same exchange number. 


Helpful Hint: Telephone exchange riumbers are 
similar to zip codes in the way that they are 
located in a specific geographic region or zone. 
Therefore, when you see two exchange numbers 
that are similar (e.g. 224 and 225), they may both 
belong to numbers which are in the same area, 
designated by the phone company. (Most phone 
books have a map or chart that help you 
determine this.) 


This is just a brief look at the way you may begin to 
locate the terminee through the phone book. 
Because each case is different and difficult to predict, 
there is no set course or steps to follow. There are 
many factors to take into consideration to insure that 


your persistence is eventually going to pay off. 


Every step of pursuit should be fully explained on the 
call record. Don’t go over old ground. Build a 
picture that can help locate the terminee efficiently. 


ABOUT INFORMATION OPERATORS 


a Always contact an Operator first when you 
have a new address for the terminee. 


a Information service is not standard from one 
locality to another, or even from one 
Operator to the next. Phrase your question 
carefully. 


a Be careful what locality you use. Some 
operators will be very restrictive while others 
will look in neighboring communities for you. 
For instance, “do you have a listing for 
Loganbach in Oakfield or any of the 
surrounding communities?” 


ABOUT THE MAILING PROCESS 


Even if you do not use a mail survey, mail contacts 
are an important support to your telephone location. 


a All mail that is sent out should travel First 
Class. 


a An attractively designed letterhead printed in 
colored ink, as well as _ colorful 
commemorative stamps, should go on the 
envelope being sent. Remember that you 
will be competing against bills and junk mail 
for priority in being forwarded and/or 
opened. The mail you send needs to be as 
distinctive as possible. 


| All envelopes should request address 
correction and forwarding so as to provide 
any new information that may aid in the 


location process. 


| The Pre-Survey Letter (PSL) should always 
have a business reply card included, so that 
the respondent can update any address or 
telephone information. Experience in one 
State has shown that over 90% of all returned 
post cards eventually result in a completed 
interview. All updated information, whether 
from a returned letter or postcard, will 
provide clues you need to continue both ail 


and phone pursuit. 


| Never assume the terminee received the first 
letter just because it was not returned to you. 


Multiple mailings help response rates 
because they stimulate call-ins. 


TIMING 


Timing is important. Allow time for post office 
changes to be returned to you before the interviewing 
window opens. Subsequent mailings should allow 
reasonable time for the telephone interviewers to do 
their job. Response rates of 50% are easily and 
quickly attained by telephone. A second mailing can 
increase your response rate by 10 percentage points. 


Examine your operational statistics to determine the 
most efficient timing tor your subsequent mailings. 


a Pre-survey letter - Send ten days prior to 
opening of interviewing window. 
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a Second letter - Send ten days after 


interviewing window opens or wait until 
response rates are at least at 50%. 


a Third letter - Send at least one week after 


mailing second letter. Response rates 
should be above 60% by this time. 


Waiting for response rates to reach a 
predetermined level such as 55% for the 2nd 
letter and 60% for the 3rd letter will help keep 
costs lower. Also, it permits adequate time 
for respondents to receive messages if they 
have not been in contact with their message 
number. 


CONTENT OF SUBSEQUENT MAILINGS 


a Second letter - Reminds terminee (if they 
have received PSL) that contact is being 
attempted. This is simply another chance to 
get the terminee to call in or return the post 
card in case they may have forgotten. 


a Third letter - Another reminder to call in. 
Include a mail questionnaire so that 
mandatory information can be supplied when 
terminee is unable or unwilling to do the 
survey by phone. Remember to include a 
postage-paid business reply envelope for 
mailing the questionnaire. 


Helpful Hint: A procedure for distinguishing mail- 
in completions from telephone-interview 
completions is recommended to deal with cases 
where contact with a terminee is made after 
receiving the mail-in survey (i.e., when a terminee 
is returning your last message call after sending 
back the questionnaire in the mail). Remember, 
as long as you have the person on the phone, a 
direct contact is more acceptable and informative 
than a written one. 


T.AG.: pp. 45-46. 
a 


While you are maintaining and collecting contact 
information by mail, you will also be attempting to 
reach the terminee by phone. A typical contact chain 


will start with the terminee’s telephone number and 
proceed to the message (M#) and then employer 
(E#) numbers where given. When these numbers 
are invalid or unable to provide any useful contact 
information, consu'ting phone books which cover the 
same communities and exchange numbers as the 
ones you are investigating can be very helpful. 


COURTEOUS PERSISTENCE 
The axiom “The squeaky wheel gets the grease” is 
very true in follow-up location activity. Each action 
should be fully documented, and always undertaken 
with courteous persistence. 


a Any address corrections and updates from 
the Post Office should be used to direct both 


your mailing and telephone efforts. 


| Any updated or verified information provided 
by the terminee on the business reply 
postcard will also guide your next step in the 


pursuit process. 


a Use of local phone books is a relatively 
inexpensive and simple way to continue 
pursult. Try to find relatives who may be in 
the same area and can provide further 
location information. Where an employer 
has been given, find their number and 
attempt to get information as well as leave a 
message. 


MOD 3, pp. 22-23. 
es 


Use of Criss-Cross directories of urban and 
suburban areas can be helpful in finding 
neighbors who may provide information 
leads and/or take a message to the 
terminee. These directories can be useful in 
verifying the phone number of a particular 
address, whether that number is unlisted, or 
whether it is listed to someone else. Also, 
the directories may be used along with a 
phone book to find relatives who might live 
near the potential respondent. (See page 31 
of this module for guidance on how to 


identify yourself.) 


JO 
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What you do next in terms of locating a terminee 
follows a statistical piogression in which you are 
guessing as to what has the highest odds of success. 
This will be different in each case. Full 
documentation of your decisions, and thelr results, 
help you make your next move. 


The number of previous attempts, the outcomes of 
those attempts, as well as your potential rewards (the 
effect on your response rates) will help you decide on 
whether a particular number is worth pursuing. %# % 


| WHO IS CALLING, PLEASE? 


There is appropriate concern throughout the system 
for the privacy rights of JTPA participants. All 
personal information collected at intake, all 
transaction information, all termination information 
and all follow-up information is therefore privileged 
data and cannot be considered public. Therefore it 
cannot be released to the public without the 


participant's expressed permission. 


It is not necessary, however, to mask the individual's 
involvement by not putting your organization's name 


on correspondence or identifying yourself in a phone 
Call. 


The following are problem areas that have surfaced. 


Can | put the name of my organization on the 
return address of correspondence to the 
terminee? 


Yes. You want to differentiate your correspondence 
from bills and junk mail. Your organization should 


have independent credibility with the individual. 
Therefore, it is more likely that the letter will be 
opened if your name is on the envelope than ff it is 


When | call the terminee’s household, how do | 
identify myself? 


Fully identifying yourself establishes your credibility 
and increases the likelihood that you will talk to the 
terminee. The ways to introduce yourself are varied, 


but this is one suggestion: 


"Hello, my name is . rm 
trying to reach [terminee’s full name] for 
the State of [State name] JTPA evaluation 
center. Is he/she available?" 


When | call a message number the terminee 
provided to us, what do | do? 


Proper identification differentiates you from a friend, 
bill collector, and other calls. The terminee should 
have notified the message number that a call from 
JTPA may come to them. 


"Hello, my name is . rm 
trying to reach [terminee’s full name]. 
He/She gave this as a message number. 
Do you know [terminee’s full name]}?" 


Respondent and message numbers don’t work. | 
call the person’s employer. What do | say? 


The primary reason to call the employer is not to get 
an interview at that time. It is to leave a message for 
the person to call your office, although you will often 
be lucky enough to get a completed interview, when 
circumstances allow it. 


Employer calls need to be handied with care. Most 
employers will be cooperative. A few will not. If the 
employer has previously hired JTPA terminees, then 
they are probably aware of the program and may 
even have been an OJT Employer or other service 
provider. 


Nonetheless, you want to provide enough 
identification to establish credibility and not so 


3 


Nothing has worked. I'm looking for relatives 
through the phone book. How do | approach this? 


This is when you need to be most cautious. Make no 
assumption that any other family members are aware 
of the person's involvement, and definitely assume 
that such knowledge would prove embarrassing to 


This is a possible introduction that you might use: 


"Hello, my name is . Tm 
trying to reach [terminee’s full name]. I'm 
contacting other families in the area with 
the same last name hoping to find a 
relative. Do you know [terminee’s full 
name]?* 


At ali times you conduct yourself as a professional. 
Operate with the terminee’s best interests in mind 
and look at things from their perspective. if you weve 
the terminee, and JTPA was calling your cousin to try 
and locate you, what would you want them to say? 
This is a very personal business. Your introduction 
and how you conduct yourself counts for a lot. 3 3 


Diliman, Don A. Mail and Telephone Surveys: The Total Design Method. John Wiley & Sons, 1978. 


| How much emphasis does your organization place on your interviewers and their skills? 
in Module 5 we look at the importance of the interviewer and the skills necessary to be a 


good interviewer. 


a What should your interviewers know about JTPA for follow-up and how do they introduce 
themselves? We provide basic background information and help your interviewers along 


with sample introductions. 


a What happens when a terminee refuses to answer questions? Several ways to overcome 


this problem and others will be discussed. 
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In this module we look at several different aspects of interviewing. We look at the importance of th interviewers in 


There are certain basic guidelines for professional 
research interviewing. We will present some basic 
guidelines that will ensure competent data collection, 
even if those now involved in follow-up have never 
had any prior research experience. 


Who the interviewer is will vary from site to site. It 
may be a person who does this for a living. It may be 
an SDA counselor. Whatever else a person does, 
when that person is conducting an interview, that 
person is an interviewer. These suggestions are for 
those persons and those who are involved in any way 


with the interviewing process. 


First and foremost is the recognition of the 
importance of the interviewer. Interviewing is difficult 
and demanding. It requires a number of skills that 
must be used simultaneously as well as effectively. 
The interviewer is the one person who will be in direct 
contact with the former participant. The outcome of 
that interaction has a direct impact on response rates 
and reflects on the State or SDA. 


[ THE INTERVIEWER IS A KEY PLAYER | ft: 


a TAG.: pp. 49, 51. 


In addition to being a recorder of information, 
observer, diplomat and researcher, the interviewer 
also becomes a confidant for every person 
interviewed. That means that all th. information 
given to the interviewer is to be held as privileged and 
confidential. Even in the course of a brief follow-up 
interview, people will share things that they will tell 
very few people, including close relatives. The 
interviewer is in a position of trust and any violation of 
confidence constitutes a breech of ethical conduct. 


Every interviewer must understand that they are 
bound by this standard. The American Association of 
Public Opinion Research has established a code of 
ethics that sets an industry standard and should be 
adhered to by all interviewers. % 9 


Fe TAG.: pp. 
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SUCCESSFUL INTERVIEWING AS A SECOND-NATURE SKILL 


Nothing of the interviewer should be reflected in the 
recorded information. Remember bias as we 
proceed with further characteristics anc! examples of 
how easily bias can occur and how to avoid it. 


a T.AG.: pp. 51, 55. 


RECORDING INFORMATION ACCURATELY 


The interviewer's job is to record information as it is 
told, exactly as it is told, without any indication of 


surprise, shock, agreement or disapproval which 
would influence the balance of the questions. 


Quite unexpectedly, interviewers could be asked for 
their opinion instead of receiving an answer to the 
question just asked. The interviewer should never 
give their own opinion. One way for the interviewer 
to handie this situation is to say, “! really don't want to 
take up too much of your time and right now it's 
much more important for me to hear what you have 
to say.” 


The interviewer's objective is to create a “warm 
neutrality’ in which the former participant feels that 


there are no right or wrong answers and that all 
responses ere acceptable. 


CASUAL AND CONVERSATIONAL TONE 


The interviewer needs to make the questionnaire 
sound as interesting and as natural as possible. The 
#1 rule is to be totally familiar with the questionnaire. 
interviewers should try practicing with a tape 
recorder. The playback can be a very humbling 


experience and instantly point out what areas need 
improving. Articulate interviewers who have 


mastered the art of the guided conversation show 
some animation without overdoing it, avoid sounding 
like they are reading, and do not stumble over words 
or phrases. Interviewers need to be aware that 
paying too much attention to the questionnaire can 
result in boring monotones. They 2re usually greeted 
with a dial tone. 


POLITE AND WELL-MANNERED 


it may sound like stating the obvious but actually it is 
not. Professional interviewers always treat each and 
every person they talk to as a special individual 
deserving utmost respect. This is especially 
necessary when the person on the other end of the 
phone is not very polite or well-mannered. Basic 
politeness can often diffuse hostility or abuse and in 
many ways contribute to an interviewer's success. 


ee TAG: pp. 53, 56. 


One suggestion to help quickly establish this rapport 
is for the intervewer to use a first and last name in 
the introduction. Using just a given name has a 
tendency to be too familiar and folksy, and it sets the 
stage for one kind of bias. The use of tities (Mr., 
Miss, etc) is too formal, creating another type of bias. 
With the former, what is being communicated is a 
need to answer in ways that wil please the 
interviewer and with the latter, that this wil be an 


interrogation. 


JU 
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GOOo LISTENER 


it may sound obvious, but a distinction needs to be 
made between listering and hearing. Each of us has 
had the experience of talking to someone who heard 
the words but was not really listening. Interviewers 
need to be alert to this. 


Let's say, for example, that a complicated discussion 
with a person who had worked all of the 12 weeks of 
the follow-up period but changed employers during 
that time has just ended. The former participant 
freely provided hourly wage information for hoth 
jobs. On the surface all appears to be clear cut until 
the time comes to enter a number for wage per hour 
for the reference week. in reviewing the 
questionnaire it is row uncertain which wage applies 
because the interviewer heard the figures but did not 
listen for which one went with the current employer. 
Although it may be possible to confidently 
reconstruct this, what if it isn't? 


ee T.AG.: pp. 54, 55. 


Potentially, you lose two ways. First, if cannot be 
corrected, the questionnaire does not count and the 
response rate suffers. Second, t can never be 
assumed that you can make another phone call and 
straighten it out. 


In many cases, especially the Title lA population, the 
one time you have the former participant on the 
phone may be the only chance you get. There may 
be no home phone or it may be unlisted. The call 
may have come in from a phone booth or the 
interviewer may have called a collateral number. in 
any case, this may be the only opportunity that the 
interviewer will have to complete the questionnaire. 


ABILITY TO DEAL WITH SENSITIVE QUESTIONS AND 
SITUATIONS 


it is well known among survey researchers that the 
three most sensitive questions ever asked are race, 
age and income. For follow-up purposes the first two 
are obtained at intake. The third, however, is the 


core ot the federal mandatory questions and no 


questionnaire can be counted that does not have 
these questions satisfactorily answered. 


ee T.AG.: pp. 49, 50, 54. 


interviewer sensitivity also plays an important role in 
the location process as well. The ability to assess 
and deal with the emotional environment at the other 
end of the telephone is never simple and can 
occasionally be stressful. A former participant may 
have been incarcerated, recently divorced or become 
seriously fl, among other things. In all instances, the 
interviewer's judgment and graciousness are called 
upon to know when and how to proceed or withdraw. 


Successful interviewing is made up of many different 
skills which need to become second nature. The role 
of the interviewer is considerably more than asking 
questions. We cannot emphasize this enough. 
interviewer Conduct is important to the accuracy and 
reliability of the data and increasing response rates. 
ideally, there will be a sense of goodwill created 
among those who participate in follow-up. 


pe MOD 1, pp. 4-7. 
a 


KNOW THE SURVEY 


Practice the questionnaire. Read it aloud. Use role 
playing. Go through it for all possible responses -— 
employed all 13 weeks, employed but did not work 
the reference week, and unemployed. Are other 
questions being asked that need to be asked of some 
people but not others? Know exactly where to go 
next. 


Pay attention to phrasing. An interesting and 
conversational tone almost always equals a 
completed interview. % 3 
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eP BEFORE YOU GET ON THE PHONE en 


Establishing credibility with terminees is one of the 
things an interviewer needs to do. One of the keys to 
this is being familiar with JTPA. When a terminee 
raises questions for the interviewer, the response “I 
don't know’ communicates a lack of knowledge 
which makes the interviewer look foolish. This could 
cause the terminee to change his or her mind about 


completing the interview. 


ee TAG:: pp. 62-65. 


it will not happen all the time, but terminees will ask 
questions about the job training structure. They may 
not know that the service or training they received 
was funded by JTPA. It is not necessary to quote the 
act but being able to provide a brief explanation is 
useful. Your response over the phone should be 
short and to the point. 


eS Understand JTPA 


Example. JTPA is the Job Training 
Partnership Act. Basically, it is a federally 
funded program that is designed to help 


a Know the populations you are going to be 
calling. Almost everyone knows by now that 


locating former participants is not easy. We 
have advocated the use of a separate 
Terminee Cali Record on which it is possible 
to include the title of the program in which 
the person was enrolied. This gives the 
interviewer a Clue in advance as to how hard 
or easy a particular person will be to find. 


Participants who were employed at the end 
of their program. A lower percentage of 
these people will still be employed than in 
Title llA-non-welfare which makes them 
somewhat more difficult to locate. 


TITLE HA - NON WELFARE — NOT EMPLOYED AT 
TERMINATION 

Participants who were unemployed at the 
end of their program. These people are the 
second hardest to locate. 


TITLe HA - WELFARE — NOT EMPLOYED AT 
TERMINATION 

Participants who were unemrixyed at the 
end of their program. These peuple are the 
most difficult to find. 


Knowing the different types of terminees alerts the 


interviewer that slightly different approaches might 
need to be used. For example, the interviewer would 


3b 
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po MOD 4, pp. 31-32. 
nee 


not ask a iocation question such as, “Is there another 
number where | can reach (name)?* on the first call 


to someone who is in an easy to find category and 


simply not available at the moment. However, that 
opportunity could present Rself on the first call to a 


Tite liA-welfare unemployed person. %* 


B 
AFTER "HELLO"...or... 
i "WHAT DO | SAY NOW?!" As 


Practice the introduction. The first 15 seconds 
following the phone being answered are the most 
important in any interview. interviewers have only 
that amount of time to establish their credibility and 
gain respondent acceptance. interviewers must 
know what they are going to say and say it 
confidently, in an upbeat, polished manner that 
almost imparts the attitude that the participant is 
expecting the call. 


Once perfected, this technique can go a long way 
toward separating this call from those of sales people 
and especially bill collectors. We have already 
recommended the interviewers introduce themselves 
using their first and last name. One reason for this 
recommendation is that bill collectors and sales 


people usually depend on first name only or a title 
and last name. 


ee TAG.: p. 53. 


_ 
To further establish credibility, every introduction 
should contain three pieces of information that are 
volunteered to the purticipant: 
a Name of the caller. 


a Who you are calling for (State, SDA, other 
organization). 


a" Why you are calling. 


There are many different ways these can be tailored 
to sult specific needs but here is one approach. 


Hi, this is (first name/last name) with 


(organization). I'm calling for 
(State /SDA). May |! speak with 
(participant name)? 


that we are doing an evaluation of 
(State's) JTPA/PIC programs for the U.S. 
Department of Labor. We would like to 
ask just a few brief questions about your 
experiences after leaving the program. 


Notice that this introduction is designed to get 
Straight to the point, with no superfiuous phrases that 
could bias the interview. We do not recommend 
using, “How are you today” as an opening line. Most 
people find this to be to be quite irritating. 


Remember, that in most cases, how the interviewer is 
initially perceived determines whether or not an 
interview will take place. This is directly linked to the 
interviewer's ability and preparation answering other 
questions the participant might have. Two that are 
most Common are: 


Q. Why is this being done? 


A. This evaluation is required of every State 
by the United States Department of Labor. 
We hope that by talking to people who have 
been involved with the programs we can find 
out how well they have worked and what 


improvements need to be made. 
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Q. Why are you Cz jing me? 


A. Every person who was involved with a 
program has a chance of being selected. It 


However each interviewers answers are tailored, 
they should be brief and spoken in non-academic 
terms. They also should be internalized, not 


is a representative, scientific sample. memorized. %* 3 
| DEALING WITH PROBLEMS | 


INTERVIEW REFUSALS 


It is possible to get most people who are reluctant to 
participate to complete the interviews. These people 
have usually had an unpleasant experience with their 
service or program. Try to emphasize that the 
reason for this interview is to find out what 
participants really think about the local programs. 
Explain that JTPA/Pi° is equally interested in finding 
out what wurked as wel! as what did not. 


Never, under any circumstances, should an 
interviewer get drawn into an argument. if the 
situation can't be diffused, thank the person for their 
time and politely withdraw. A second call on another 
day and another time may be greeted more 


hospitably. 


as T.A.G.: p. 56. 


WAGE INFORMATION REFUSALS 


We have already said that asking any questions 
about a person's income is one of the most sensitive 
areas of inquiry. If a participant is going to balk on 
any section of the questionnaire, more than likely it 
will be here. The response can be treated very 


simply. 


The focus should not be on the confidentiality, 
although that can be mentioned, but on the purpose 
of the question. 


Congress, which has funded JTPA, wants to 
know if the program is doing its job. They 
want to find out if there is less reliance on 
welfare, if people are getting jobs, and if 
these jobs are above minimum wage. One 
of the ways to measure this is to ask former 
participants about their earnings. 


Stating that all information is reported in combined 
form with no linking between individuals and their 
answers is sometimes sufficient to reassure 
terminees into completing the interview. The 
interviewer needs to understand this in order to 
expiain it to former participants. 


“LGOT MY JOB ON MY OWN" 
The possibility exists that this statement could be 
heard from virtualiy anyone. 


The interviewer can say that even though that is the 
case, we are contacting people who have had an 
involvement with JTPA funded programs whether or 
not they completed the program and whether or not 
JTPA was responsible for them getting their job. This 
is one way of comparing wage and income levels 
among people who got jobs through the JTPA 
system and those who got their jobs on their own. 


PROBING 


In addition to asking the appropriate questions of the 
former participant, the second largest area for 
interviewer error is probing. 


Probing is the use of careful, conversational and, 
most importantly, neutral and non-directional 
Statements in order to elicit a response that is 
appropriate to the question being asked. 


a T.AG.: pp. 54-56. 


The interviewer must instantly recognize when an 
answer is insificient in terms of the question 
objectives, then be prepared to probe, sensitively 
and, appropriately, to encourage a more specific 
response. Unbiased probing cannot be 


SY 
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accomplished unless the interviewer is totally familiar 
with the questionnaire and its instructions. 


Four kinds of probes are commonly used in survey 
research. They are easy to learn and can be valuable 


for ensuring the accuracy of the data being collected. 


a Repeating the Question. By repeating the 
question, exactly as written in the 
questionnaire, the interviewer provides the 
respondent a graceful opportunity to answer 
a question that may not have been 
understood. Changing the emphasis on 
particular words can often help in 
communicating the meaning of a question. 


a Repeating the Answer. Immediately 
repeating the participant's own words is very 
effective in encouraging elaboration on an 
otherwise unclear answer. The mechanics of 
hearing the words repeated may result in the 
addition of a meaningful comment which 
might otherwise be lost. Don't paraphrase. 


INFINITE DIVERSITY - 
| WHEN IT DOESN'T FIT THE MOLD 


a The Expectant Pause. Timing is everything 
in using silence as a way of furthering 
comment. it can communicate that the 
interviewer is interested in and waiting to 
hear more of what he or she has to say. 
However, be alert to the possibility that there 


is actually nothing left to say. 


| Asking for Further Clarification. Asking a 
participant to elaborate has many benefits. It 
devnonstrates that the interviewer cares 
complete. This in turn establishes a rapport 
and a trust that sets the tone for the 
remainder of the interview. It enhances the 
reliability of the data. 


SUGGESTED NEUTRAL PROBES. 
a Which number do you think comes closest? 


a If you had to choose just one of those, which 
would it be? 


f Of the reasons you mentioned, which is the 
most important? 


Probing has a very practical application in terms of 
the mandatory wage information questions. We need 
to be able to calculate gross w. ekly earnings for the 


Title IIA participants and average hourly wage for 
EDWAA participants. 


As straightforward as asking these questions may 
appear on the surface, this is not always the case. 
Here are some examples of responses interviewers 
will be faced with and suggestions for handling them. 
Working over 40 hours. 


a Was the participant paid time and a half for 
the hours over 40? 


L Was the person working more than one job? 


it has happened that a person worked their basic 
number of hours and their overtime was paid at both 
time and a half and double time. Those hours can 
also be paid at differential rates for different job 
functions. 


Long distance trucker. 
| Was the person paid by the load or by mile? 


a Was the person paid a flat hourly rate plus a 
load or mileage bonus? 


Wages for long distance truck drivers are 
complicated. The interviewer needs to be alert when 
a person says, “! don't get paid per hour." The skilled 
interviewer should then ask the respondent if he or 
she is paid mileage or by the load. Ifthe respondents -, 

5 
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share their gross salary for the week it is still 
important to get a clarification of how they are paid. 
When respondents tell you they get paid by the mile 
coliect both the number of miles driven and the 


mileage rate. 
Working, but not in the reference week. 


a Was the person on vacation or sick leave? 


| Was the person paid for the time off in the 
reference week? 


Many companies offer paid sick leave or paid 
vacation to employees. If the respondent is working 
for a company that offers this benefit, then the 


amount of money received for this time is important 
to record. 
Working but not receiving money. 


a Was the person working for a commission? 


A person working in sales may be paid totally on 
commission. They might “receive” income for a 
previous week's sales, though they “earned” no 
income in the Reference Week. We need to capture 
“earnings” for the Reference Week even though the 
individual might “receive” these funds later in the 
month. 


Some comoanies reimburse their sales staff for 
expenses, but these payments are not income. 


Self Employed 
| Was the respondent doing things to help the 
business but had no calls or sales? 


By highlighting some of the situations that 
interviewers need to know how to handle, we once 
again want to draw attention to what was said at the 
beginning. How skilled and prepared the people on 
the phone are is the very first step in assuring the 
quality of the data and the reliability of any reporting 
and analysis that would follow. 3 3 


Gorden, Raymond L., interviewing: Strategy, Techniques and Tactics, The Dorsey Press, 1975. 


1991-1992 Directory of Members: AAPOR, P.O. Box 1248, Ann Arbor, Michigan, 48106 (313) 764-1555. 


Rich West is the statistical contractor for USDOL. For the really detailed questions, you can contact at: 
Mr. Rich West, Vice President; Social Policy Research Associates; 200 Middlefield Road, Suite 100, 


Menlo Park, CA 94025; (415) 617-8627 


| In Module 6 we will begin to make the connection between the collection of postprogram 
data and the utilization of that data for program assessment, monitoring and management 


purposes. 


a New and different options to help you analyze your data will be discussed. 
a Now that you know what sampling is and how to do it, why do you do it? This question 


will be addressed. 


a Are there any problems with “Spring cleaning” your MIS of inactive participants? We'll 


show you things to look out for. 


a How do other States organize their follow-up? We'll show you the options other States 


selected. 
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T.A.G.* has two major concerns guiding the selection of materials. The first concern is with the technical 


discuss other evaluation research options you have. Another section reviews confidence issues related to 


sampling your population. Also discussed are the problems with spring cleaning your data files. And finally, we 
look at how States are organized for follow-up. 


| DATA LINKAGES | 
The amount of data available to JTPA is enormous. financial, work, economic and social background are 
When analysis of data is considered, too many recorded. 
people focus on the follow-up data elements. The 
horizon is much broader. Birthdate 
Ethni 
Everyone has data from at least four different F vanes 
sources. Any site involved in asking additional Sex 
postprogram evaluation questions beyond collection Family Status 
of the USDOL mandatory questions has a fifth source Welfare Status 
of data. Amount received 
Lack Si nt Work History 
The examples which follow are illustrative only. recites ork om 
Standardized Participant Informution Reporting will Ex-Offender 
not necessarily require all of these variables. Title of Participation 
Employment Status 
INTAKE Hours/wage 
The first source is intake data. This is one of the DATA SOURCE ONE 


most important components of your MIS. This is the 
point at which many elements of an individual's 


GF TAG: pp. } 22 | MOD 
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The second source is the program participation data. 
In this component, the variety of services received by 
the participant are recorded. Length and cost of 
service, successful completion, vendors providing 
Services, and other variables may also be collected. 


Activity Type 

Start Date 

End Date 

Completion (Status/Code) 
Cost of Activity 

Vendors 

Counselors 


DATA SOURCE TWO 


TERMINATION 


The third major source of data is the termination 
data. When ail is said and done, this component 
records the bottom lines -- employment information, 
termination status, employment wage, type of job, 
and so on. 


Termination Code 
Termination Status 
Termination Date 
Employment Code 
Employment Wage 
Hours Per Week 
Job Title 


DATA SOURCE THREE 


FOLLOW-UP 


The fourth source will be available on all or some 
terminees depending on whether sampling is used to 
select terminees for the follow-up survey and whether 
or not the survey was completed. This component 
will contain the mandatory data items required by 


USDOL -- weekly earnings, hourly rate, weeks 
worked of 13. 


Employment Status at 13th Week 
Weekly Earnings 
Number of Weeks Worked During 
Follow-Up Period 
Hourly Rate of Pay 
EDWAA recalculated 


DATA SOURCE FOUR 


pe MOD 3, pp. 21-22. 
AS 


EVALUATION 


States and SDAs which ask additional questions of 
the terminees will have postprogram evaluation 
information. This will obviously vary by site 
depending on the questions asked. As discussed in 
Module 3, operational statistics from the location 
process can also be available. 


Welfare Status 
Amount Received 
Satisfaction with Program 
Employer Benefits 
(Current/Continuing) 
Barriers to Employment 
Number of Contact Attempts 


DATA SOURCE FIVE 


Linking these four or five major sources of data 
together creates a record that contains vast amounts 
of untapped information. The number of questions 
that you can explore is almost limitless. To benefit 
from follow-up, you must ask good questions. This 
comes up first as you design a good postprogram 
evaluation survey. 


it comes up again as you face the quantity of data 
available to you, and sort through your analysis 
priorities. What is it you really want to know? Given 
your limited rescurces for such analysis, what Is your 
first priority of knowedge? 


Once you have your data linkages, consider the 
ideas in the next section. % 
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WHAT ! ALWAYS WANTED TO KNOW ABOUT MY JTPA PROGRAM, 
BUT WAS AFRAID TO ASK 


The following list is offered to stimulate your thinking 
about how to translate data into useful information. 


T.AG.: pp. 8-11. 
ae 


Are benefits more important than wages for 
job retention? 


Does education level affect employment 
Status and earnings? 


Has welfare dependency decreased since 
program intervention? 


Of those individuals still receiving welfare 
benefits, has the dollar amount increased 
or decreased? 


Does the length and type of program 
intervention affect employment retention? 


Can we estimate a “return on investment?" 


Are my response rates influenced by the 
number of contact attempts? 


Is there a disparity in the type of services 
received between and among program 


populations? 


Are OJT participants being retained by 
employees after the wage reimbursement 
has ceased? 


There is literally no end to the number of good 
questions you can ask. Which of these questions 
can be answered, however, Cepends on your data 
linkages and organization, as well as your survey 
design. You can't analyze questions you didn't ask. 


lt is only when you begin analyzing your data that 
you discover what does and does not work and 
where your information gaps exist. 


You may find that you want to know more than just 
what the 13 week survey tells you. 9% 


THE LIGHT AT THE END OF THE TUNNEL -- 
LOOKING DOWN THE ROAD 


LONG TERM EVALUATION 


The 13 week postprogram survey really constitutes a 
short term assessment of the success of the JTPA 
program and intervention. Many States have felt the 
need for longer term measures. 


For instance, Kentucky, Massachusetts and Texas 
conduct 52 week surveys. Ohio conducts a 65 week 
survey. New Hampshire conducts a 78 week survey. 
Other States may also be conducting long term 
assessments. 


Long term evaluation does not have to be a huge 
expense. Once you have met the 13 week follow-up 
requirements, you can do what you want in terms of 
longer range research. 


The 13 week survey must follow specified sampling 
requirements. Anything else can be done to meet 
your specific needs. For instance, if you are a State 


which samples terminees, you can select a sample 
from all SDAs that is representative of your Statewide 
terminee population. You don't have to interview 
everyone at 52 weeks that you interviewed at 13. 
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EMPLOYERS 


You might want to conduct a special survey with 
employers who have hired your terminees. How 
satisfactory are your training programs from the 
employer's perspective? What types of training 
would be more beneficial? 


WaGE RECORD 


You may want to link your data with the 
Unemployment insurance data base (UI) to look at 
much longer term workforce retention issues. You 
can review quarterly data for a period of two and a 
half years or more in some States. 


SDA/PIC 


Surveying the SDA staff and PIC representatives can 
provide you with insights to priorities and 
perspectives. The view of JTPA can be very different 
depending on whether you are a participant, PIC 
chair, or a counselor. 


VENDORS/CONTRACTORS 


Vendors and contractors are yet another part of your 
service delivery system. The opportunity for 
systematic feedback may be welcomed and yield 


some new insights into the difficulties as well as 
opportunities that these people see. 


LONGITUDINAL ANALYSIS 


Even without expanding your data collection, you 
have the ability to conduct longitudinal analyses. 
What has happened in your program over the last 
four years? What changes have you seen? There is 
a great need for longitudinal learning in the system. 


In other words, consider what you want to find out 
and what you will do with it. While financial 
resources are always a concern, many options exist 
to creatively and innovatively collect longer term 
assessment information that can be of value in 
evaluating your programs. 


pe | MOD 1, pp. 2-3. 
es 


Module 1 contained a chart of the JTPA system. 
Useful planning and assessment informai'c:: can be 
gathered from all meinbers in the system. It is 
important that you don't just do it. Do it in order to 
use it. 3k 


| _ NEW ANALYTIC OPTIONS | 


One of your new sources for information will be the 
SPIR (Standardized Participant Information 
Reporting). With the advent of the SPIR, many new 
analytic options will be opening up. As you should 
be well aware, however, the amount of data 
submitted to this system is but a tiny fraction of the 
data you have been collecting for years. 


The SPIR ever in no way limits you from 
continuing to oc <.wct your own tailored analysis of 
the data you collect. The future of SPIR and its 
eventual analytic capabilities is still being discussed 
and designed. Interstate analysis of standardized 
data will soon be a reality. This will create new 
opportunities for analysis. 


You, however, will always be in the best position to 
determine what your particular information needs are. 
SPIR is merely an expansion and extension of what 
you are able to do today. Both systems have great 
potential. 3 
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All aspects of the data collection process, from filling 
out the application through follow-up interviews are 
important. When you get to the point of analyzing 
your data, you are in a position in which you must 
trust what someone else has recorded. 


With respect to the follow-up and posiprogram data, 
a key variable in your level of comfort and confidence 


is the sampling process used. 


T.AG.: pp. 25-37. 
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Sampling is based upon statistical theories of 
probability. A true random sampling procedure gives 
everyone in the universe an equal chance to be 
selected. 


The sample is selected at the SDA !evel. Because 
one emphasis of the JTPA program is on reducing 
welfare dependency, a supplemental sample of 
persons on welfare must also be chosen, if 
necessary. The only case in which a supplemental 
sample is not necessary, is when the SDA terminates 
less than 138 persons altogether in the Program 
Year. 


All of these sarnples are chosen by a formula based 
on the number of planned terminations an SDA will 
have in a program year. You can find this chart in the 
1990 T.A.G., on page 25 as well as in the instructions 
for the JASR and WAPR. 


Confidence problems emerge in three cases. 


CASE ONE. Your plan might change. if you 
terminate more people than planned, then your 
confidence will increase because your effective 
sampling percentage will be larger than necessary. 


Conversely, if fewer people are terminated than 
planned, your level of confidence will decrease 
because your sampling percentage is too small. 


SAMPLING — AN ISSUE OF CONFIDENCE 


MOD 7, pp. 51-55. 
ee 


CASE TWO. Module 7 discusses late terminations in 
some depth. For now, be aware that some SDAs do 
not have a procedure to ensure that vendors are 
reporting terminations in a sufficiently timely fashion 
to be included in the sampling process. 


Tracking late terminations should also be a State 
service to SDAs to help them ensure timely reporting 
of key data. Depending on what percentage of your 
terminations are reported after sampling occurs, the 
planning data you receive from the follow- 
up/postprogram evaluation system may not be 
accurate. 


CASE THREE. You have low response rates. 
Response rate problems fall into the technical 
category of “non-sampling error.” What this means is 
that the people you cannot locate and interview may 
be different in many ways from the people that you 
do locate and interview. Low response rates mean 


that your data do not accurately represent your 
sample of terminees. 


T.AG.: pp. 7, 71-74. 
Eee 


While USDOL has specified a minimum 70% 


response rate for all groups, the higher the response 
rate you can achieve, the more confidence you can 


have that your data fairly represent the experiences 
of all your terminees. 


Study your own situation and ask yourself how well 
your system nm easures up. 


a How accurate was your plan compared to 


actual terminations? What caused the 
variation? 


& What percentage of terminees were reported 
too late to be sampled? Is there any pattern 
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by vendor, type of service, or demographic 
variable? 


| is there any pattern; by vendor, type of 
service, or demographic variable; among 
those persons we cannot locate and 
interview? 


Linking and studying your data has to be done with 
increasing caution as your confidence in your data 
decreases. % 3 


PROBLEMS WITH SPRING 
oie | CLEANING YOUR MIS ! oie 


Wouldn't it be perfect if all persons who came to the 
JTPA system for assistance resulted in positive 
terminations? Reality, however, does not provide us 
that perfection. For whatever reasons, you will not be 
able to help everyone. 


There are very specific rules to be followed 
concerning just how long termination can be 
postponed. This is called the “inactive” status. 
Participants who have received training may be kept 
in inactive status 90 days before a termination must 
be claimed; inactive status is 30 days for those 


receiving services only. 


Sites should typically “clean” their cases in the MIS 
on @ quarterly basis. That is why a rise in negative 
terminations can be expected during certain weeks of 
the year. However, we want to avoid having an 
enormous negative termination rate in the last week 


or so of the program year. 


One State which has looked at these patterns 
discovered the following. The last week of the 
Program Year is 2% of the year. The State found that 
3% to 8% of its positive terminations for the year 
occurred in this week. It also discovered that 12% to 


| CONDUCTING 


T.AG.: pp. 17-20. 
eel 


From PY87 to PY91, very littie has changed. Some 
States have experimented with different options, but 


69% of its negative terminations (depending on title 
of service) occurred in this week. 


The impact of this pattern of termination is that 
negative terminations disproportionately appear in 
the fourth quarter of the Program Year. These 
terminations also have the oldest location data which 
makes them the rnost difficult to locate and interview 
for foliow-up. 


The fourth quarter of the Program Year is also the 
first quarter of the JASR/WAPR follow-up reporting 
and performance standards year which means that 
such skes are starting the reporting year with the 
worst possible results. 


From a planning and assessment perspective, having 
negative information properly integrated as it 
naturally occurs throughout your program year 
provides the greatest benefit in identifying the 
strengths and weaknesses of your program activities. 


Proper and timely termination of your participants 
allows you the reaction time you need to fine-tune 


and adjust your program. % 


UR FOLLOW-UP 


most have remained with the option selected at the 
time follow-up was implemented. 


While the 1990 TAG. reported the organization 
styles of States, our table also indicates the 
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approximate percent of SDAs under these different 
systems. 


SDA-collected data, whether in-house or contracted, 
is Clustered under decentralized. For PYS1 we show 


the approximate percentage of SDAs under each 
method 


Number of States* by Type of Follow-up System 


PY87 | PYS1 | %SDAs 
Centralized State 
in-House 12 10 11 
Centralized State- 
Outside Contractor 24 23 §3 
SDA Decentralized 16 19 3% 


* includes District of Columbia and Puerto Rico 


States that have the same follow-up options may also 
share common concerns. Because networking 


among your colleagues is always encouraged, we list 
the States and the PY91 follow-up option used. 3 3 
DECENTRALIZED/SDA COLLECTED 


Arizona 


Tia 


CENTRALIZED/SELF COLLECTED 
Alabama New Mexico 
District of Columbia North Carolina 
Massachusetts North Dakota 
Missouri South Dakota 
Montana Wyoming 
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Applied Seymour Sudman, Academic Press inc., 1976. Most books on sampling are 
technical. one of the more readable. very 


Social Statistics, Hubert M. Blalock, Jr., McGraw-Hill Book Company., 1979. This is considered one of the 
bibles of social research statistics. 


Social Science Data, Frank M. Andrew, et al, 
of Michigan, 


1 University Ann Arbor, Mi. This is a 
and has helped many a researcher wade Gwough the morass of proper sstection and 


Handbook of Political Science Methods, G. David Garson, Holbrook Press, inc., 1971. This is only one of a 


variety of elementary handbooks that explain how to properly use 
: A Modest introduction, Dennis K. Benson, Appropriate inc., Columbus, OH. 
bret mafograph was writen for the layperson. explains targe and smal sampling and how to Compute 


@ We begin to look at the utilization of postprogram data that has been collected. 


| Who are the audiences for postprogram data and what are their interests? 


a Quality control procedures should be in place throughout the follow-up system. We will 
show you what to look for and why. 


| Late terminations can introduce non-random bias into your postprogram data. Learn how 
this can happen. 


YY 


T.A.G. + 


Supplement to the Technical Assistance Guide for JTPA Follow-up and Validation — MODULE 7 - 1982 


Published by the Employment and Training Administration, U.S. Department of Labor, Fin NS629, 200 Constitution Avenue, N.W., 
Washington, D.C. 20210 - (202) 219-5782 


We look at the potential audiences for postprogram data and system-evel quality control. We look at the quality 
issue of late reporting of terminations in some depth. And we begin focusing on Level 1 utilization with a quiz on 


completion of the JTPA Annual Status Report (JASR) and the Worker Adjustment Program Report (WAPR). 


Be | UTILIZATION -- UNLEASHING THE LIONS Yet 


The first six modules of T.A.G.* focused on the 
postprogram data collection process and its 
importance in an evaluation system. Over the next 
few modules, we return to utilization of the data 
collected. 


Utilization begins with identifying the informational 
needs of prospective audiences. Questions to be 
answered include: Who will use the data collected? 
What will they use it for? How will they use it? What 
specific information will they need to know? When 
will they need to know it? What are their informational 


priorities? That is, of all the things they would like to 


will not use it, thereby making it useless. 9% 9 


AUDIENCES FOR FOLLOW-UP DATA 


Public programs operate in a fishbowl. Their 
business is everyone's business. in the short run, the 
ability to access and share timely and meaningful 
information about program activities and outcomes 
helps public programs gain the cooperation of others 
and manage their own affairs efficiently. in the long 
run, it can help ensure their survival. 


Among public programs, JTPA is unusual in its 
emphasis on outcome - as opposed to process - 
measures. The fact that we do know what happens 
to our participants gives us a real advantage. 


Fe TAG.: pp. 


information in the 1990 T.A.G. is "keyed" by this symbol. 


i 


Information In TAG * Modules le flagged” by this symbol 


Prepared for the U. S. Department of Labor, ETA by Appropriate Solutions, inc., Columbus, Ohio. Contract @99- 1-3654-72-0786-01 
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in addition, because our “customers” are contacted 
when they are no longer dependent on th. program 
for services, the follow-up process has an inherent 
credibility. This credibility is maximized when follow- 
up is carried out independent of those with the 
greatest interest in the results. 


There are four types of audiences for postprogram 
data, and each exists at the federal, State, and local 
levels. They are members of the public, cooperating 
agencies and programs, oversight bodies, and 
program staff. Each group Is likely to be interested in 
differert information, communicated in different 


JTPA reaches the public through interest groups and 
the media. From a public relations perspective, the 
most useful postprogram measure is probably 
workforce retention, if currently available - the 
percentage of people work.ng at follow-up among 
those who were employed at termination. 


A staternent like: 


"80% of the people who are employed whi.n 


they leave the program arn working 13 
weeks later’ 


is easy to understand and helps to explain program 
quality. Press releases, marketing materials, annual 
reports, and public forums offer outlets ‘or 


communicating postprogram data to the public. 


Coordinating agencies are concerned about how 
JTPA-4tunded activities f in with their own agendas. 
The postprogram data which are most important to 
them depend on thelr program goais. 


For example, welfare agencies may be especially 
interested in earnings, while educational agencies 
may be more interested in occupational information. 


In general, however, information on the postprogram 
outcomes experienced by thelr clientele can help 
other programs assess the value of cooperative 
efforts. 


OVERSIGHT PF” 21ES 


Oversight bodies, which include Congress, the U.S. 
Deparsnernt of Labor, State Job Training 
Coordinating Councils, and Private industry Councils, 
are importart audiences for postprogram data. 
Basically, they need two types of information: 1) 
routine reports of key indicators; and 2) special 
analyses of data relevant to decision-making. 


Postprogram data may be used to make program- 
shaping decisions. it is important that oversight 
bodies be educated, over time, about the follow-up 
process, the information & produces, and how that 
information can - and cannot - be used. 


For program staff, postprogram data offer a wealth of 
information for planning, performance management, 
and program evaluation. While simple, routine 
reports are heipful, they cannot begin to address the 
variety of day-to-day questions which postprogram 
data can help program staff answer. Therefore, the 
degree to which staff are able to access and 
manipulate these data has a substantial effect on the 
extent to which the data can be used. % 
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a 


Program managers concerned with utilization of 
follow-up data need to ensure that the quality of the 
data collected justifies ts use. Regardless of whether 
follow-up is being done in-house, by service 
providers, or by an independent survey contractor, 
someone needs to be responsible for quality control. 


TAG: pp. 87-88. 
EO 


Quality control goes beyond ensuring that every 
response is accurate and consistent with all other 
sources of information on the same subject. it siso 
means ensuring that the system as a whole is 


operating smoothly and producing valid data. 


Two types of “information” are helpful in carrying out 
quality control - operations indicators and 
“exceptions” indicators. 


First, a set of routine indicators can flag changes in 
system performance. Such indicators become more 
useful over time, as experience provides the staff 
responsible for quality control with a sense of what 
can be expected. Some useful indicators are: 


5 The percentage of sample terminees who 
are contacted and who respond to all of the 


required USDOL questions. 


Any contact rate below the federally mandated 
response rate of 70% is problematic, but a decrease 
in the SDA's normal contact rate should be checked 
Out. Is the decrease attributable to problems with the 
quality of contact data? Does i refiect a change in 
the population being served? Some things may need 
investigating. 


TAG: p. 7. 
el 


iw 


For example, is a drop in response rates due to an 
increase in service to homeless participants and, ff 
$0, what can be done to increase the chances of 


reaching members of this group? 


po | MOD 3, pp. 22-23. 
ee 
REFUSAL RATE 


a The percentage of terminees who, when 
contacted, refuse to participate in the 
survey. 


The refusal rate should be very low -- not much more 
than 1%. If it is higher, it is important to find out why. 


For example, has a service provider discouraged 
participants from cooperating? |s an interviewer 
tailing to treat participants with courtesy and respect? 
Are interviewers well trained in explaining the value 
and importance of participation in the follow-up 
evaluation? 


a The percentage of terminees whose 
termination records were submitted 10 late 
to be included in the follow-up process, 
divided by the total number of terminees for 


the same period. 


Termination records may be selected for follow-up on 
different time schedules. Therefore, it is extremely 
important that someone is responsible for checking 
the late termination rate. 


Late terminations will affect your data whether you 
perform follow-up with all terminees or just a sample. 
Do not assume that records are being reported in a 
timely fashion. This may not be the case. 


Late terminations are very likely to inject non- 
sampiing bias into your analysis. if that happens, it is 
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important to find out where the late terminations are 
coming from and to take corrective action. 


The key to using any of these indicators is checking 
them on a regular basis and looking further when 


they suggest that a problem may be developing. 


For example, late terminations and refusals may be 
associated with particular service providers. 
Similarly, an unwillingness to provide information on 
earnings may be more common among participants 
receiving public assistance. in order to find out, staff 
responsible for quality control need to be able to 
calculate meaningful quality control indicators for 
participant subgroups. 


"EXCEPTIONS" INDICATORS 

Management By Exceptions (MBE) is a recognized 
business operating system. An “exception” is what 
you define it to be, but in general it is a discrepancy 
between something expected and something 
experienced. 


There are three essential ingredients for making 
effective use of exceptions. They are: 


1) A set of guidelines to help determine what 
constitutes an exception; 


2) A mechanism for recording and reporting 
the occurrence of an exception; and 


3) Someone with clear responsibility for 
following up on and resolving exceptions. 


Exceptions may arise within your formal survey. For 
instance, interviewers might be required to flag 
interviews with reported weekly earnings outside of 
an established, normal range and to attach 
explanations for the unusual earning level. This 
requirement has the immediate effect of making an 
interviewer stop and confirm the unusual earnings 
with the terminee. 


The flagged interviews are likely to involve unusual 
situations such as people who work on comm. ssion, 


self-employed people, and regulariy-employed 
people for whom the target week was, in some way, 
unusual. in this case, routine review of the flagged 
interviews should result in clarification of how to 
handle unusual earnings and greater consistency in 
how they are handied over time. 


HELPFUL HINT: Because the interview process 
invc'ves Contact with participants, effective use of 
exceptions can contribute to programmatic 
quality control, too. 


For example, one of the follow-up contractors 
interviewed reported that a routine procedure for 
reporting terminee complaints had identified 
employers who were firing OJT participants as 
soon as the training subsidy ended. This 
information enabled staff in the SDAs where the 
problem was occurring to put a stop to the 
practice. 


Other problems which may surface during follow- 
up interviews include incompetent instructors, 
service providers who fail to keep Commitments 
to participants, and ‘participants* who have not 


received any services. 


Exceptions may also occur when terminees volunteer 
informaiion. This may take the form of a complaint 
(against specific staff people, an interviewer, or a 
service vendor, or a program) or a compliment 
(commending individual effort or the quality of a 
service). 


Although these exceptions may not be specifically 
sought, they provide insight into your programs and 
can become opportunities for improvement. 
Negative exceptions must f« researched and 
resolved. Positive exceptions rnust be shared. 


It is important to keep “exceptions” in perspective. 
Interviewers who report exceptions have only the 
participants’ viewpoints at hand. This is but one tool 
available to SDAs and States in constantly evaluating 
the responsiveness and success of their programs. 


ok ok 
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| TEST YOUR REPORTING KNOW-HOW | 


Ultimately, the mandated follow-up process boils 
down to a few boxes on the JASR and the WAPR. 
These data are part of the information on program 
outcomes that are used and disseminated by the 
U.S. Department of Labor. Moreover, they are the 
basis for setung postprogram performance standards 
and oeveloping adjustment models. Bad data can 
translate to bad standards and models which oo not 
make adequate adjustmerits for local conditions. 


TAG: pp. 71-86. 
Des 


Do you know how to report your postprogram 
outcomes? Take the following quiz, and then check 


the answers on page 55. *%* 


An SDA expects to have 600 adult terminees, including about 120 adult welfare recipients. It plans to 
use a 50% sample for its adult data, but must attempt to contact all of its adult welfare recipients. 
Approximately how many participants altogether does the SDA expect to try to contact? 


. | An SDA had an adult sample of 400 cases. It reached 320 people. Four people refused to participate 
in the survey, and 16 people who worked during the 13th week after termination refused to provide 
information on their earnings. What was the SDA’s adult response rate? 


Using the SDA described in question 2, 250 of the people who were reached had worked during the 
13th week after termination. Total reported earnings in the 13th week were $51,240. What was the SDA’s 


average weekly earnings at follow-up? (Assume that no adjustment for response bias is needed.) 


. An SDA's adult terminee population included 300 people who had been placed and 80 people who 
had not been placed. Completed interviews were obtained for 240 of the people who had been piaced, 
and 180 of these participants were employed in the 13th week after termination. Fifty of the people who 
had not been placed had completed interviews, and 15 were employed in the 13th week after 
termination. What was the SDA’s adult employment rate at follow-up? 


. An SDA found that, among adults with completed interviews, 15% had not worked at all during the 13 
weeks after leaving the program. The remaining 85% had worked an average of 10 weeks. What was 


the SDA's average number of weeks worked in the follow-up period among adults? 


| MISSING PERSONS: LATE REPORTING OF TERMINATIONS | 


One of the most serious problems found during the 
technical assistance site visits was late terminations. 


Late terminations involve participants who are 
excluded from the follow-up process because their 
terminations are reported too late. Although this 
problem can occur in any setting, it is more likely to 
occur in centralized systems, where terminations are 


processed locally but terminees are selected for 
follow-up at the State level. 


Here's how it happens. Terminees are typically 
identified for follow-up 9 to {2 weeks after termination 
depending on State or SDA policy. Terminations 
reported and entered into the MIS after the sample is 
identified cannot be included in follow-up. 
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For example, people with termination dates of 
September 6 through 12 may be selected for follow- 
up some time during the week of November 15. If a 
termination dated September 9 is not reported until 
November 16, that participant will never be included 
in the follow-up process. 


if late terminations are a rare and random process, 
they do not have much effect on postprogram 
outcomes. However, in SDAs where significant 
numbers of terminees are “missing persons”, late 
terminations are likely to occur because of how 
certain types of terminations are routinely handled. 


For example, a service provider may postpone 
terminating a participant who has dropped out of 
training in hopes that he or she will come back. 
Eventually, the provider completes a_ termination 
form, back-dating the termination to when the 
participant disappeared. When this occurs, 
participants who were not employed are under- 
represented in the follow-up sample. 


But it can work the other way, too. Where payments 
are based on employment and retention, service 
providers may not report a termination until ail 
employment and retention benchmarks have been 
achieved. Then they use the employment date as the 
termination date. 


That strategy gives them an opportunity to try again 
with a participant who is not retained, but it can also 
mean that participants who were employed at 
termination -- the SDA’s successes -- are 
systematically excluded from follow-up. 


In either case, such follow-up outcomes are biased 
and really cannot be used beyond the individual level. 
That is, the only follow-up data which can be used 
with confidence are the individual responses of those 
terminees who were contacted. Nothing can be said 
about outcomes at the SDA, prcject, or target group 
level. 


That fact has important implications for performance 
management, program evaluation, calculation of 
performance relative to standards, and allocation of 
incentive awards. It is in everyone's interest to see 
that the incidence of “missing persons” is minimized. 


Do not simply make the assumption there is no late 
termination problem without checking your data. 
One SDA which confidently said this was not a 
problem checked its data and found that 40% of its 
terminees were reported too late to be included in 
follow-up. 


IMPLICATION ONE. if an SDA had 400 planned 
terminees, it would have a minimum sample size of 
240 (60%). If only 240 terminees were reported on 
time, only 144 people would be sampled, well under 
the minimum sample size. 


IMPLICATION Two. If the SDA also only had a 70% 
response rate, only 101 interviews would be 


completed. Instead of making decisions about its 
program and its impact based on 60% of its 
terminees, the SDA is making these decisions based 
on only 25% of its terminees. The ability to use this 
data with confidence in program planning and 
evaluation has been severely degraded. 


There are several things that States and SDAs can do 
to avoid late terminations: 


| SDAs can make sure that their own policies 
are not likely to generate “missing persons” 
and that their service providers understand 
the follow-up process and their 
responsibilities within that process. 
Disincentives for late submission of 
terminations can be built into service 
provider contracts. 


| SDAs can monitor participants beyond the 
planned ending dates by checking inactive 
and employment statuses and routinely 
asking service providers to account for 
participants who should have completed 
training but who have not been terminated. 


| States can monitor late terminations by 
comparing data entry dates to termination 
dates. Actions which may help States focus 
attention on this problem include publishing 
SDA comparative reports on the incidence of 
late terminations. States and SDAs can 
devise equitable sanctions polic‘es for late 
termination situations. it is far more 
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important, however, to eliminate the problem problems of inconsistency between what participants 


than merely to punish ft| These would be and their actual records say. 

internal policies and not related to USDOL 

performance standards. TAG: p. 72. 

es 

a States, SDAs, and service providers can go 

through a periodic participant data closeout Follow-up data can be included in or linked to other 

process which identifies and weeds cut files. However, process indicators -- simple yes/no 

participants who, although no longer fields indicating the following steps have been 

participating, are being carried as current accomplished for each case - need to be included to 

participants. help programmers and users select the right records 

for use in reports and analyses. The steps in the 

In some States and SDAs, follow-up data were process which need to be marked include: selection 
maintained in a separate file and never linked back to for follow-up, inclusion in the follow-up sample, 
application, enrollment, and iermination data. contact, interview, and “completed interview’ in 
Isolation of follow-up data limits its usefulness and accordance with the definition established by the 
makes it more difficult to detect late terminations and USDOL. 


Answer Key: Reporting Quiz 


1. Approximately 360. The adult sample of 300 will include about 60 welfare recipients, and the SDA will 
need to attempt to contact the remaining welfare recipients as well. With only 120 welfare terminees, all 
must be included. 


2. 75.0%. Only respondents who answer ail of the required questions can be included in the SDA's 
response rate. None of the data from incomplete interviews is reported on the JASR and the WAPR. The 
response rate is 300 valid surveys divided by a sample of 400. 


3. $219. Although 250 respondents said that they were placed, only 234 of those participants (250 minus 
16 refusals) had complete interviews. Divide total earnings (51,240) by total employed and reporting 
earnings (234). Earnings should be rounded to the nearest whole doliar. 


4. 65.5%. Although 195 (67.2%) of the terminees with completed interviews were employed during the 
13th week, the difference between the response rates of those who were placed (80.0%) and those who 


were not placed (62.5%) exceeds 5 percentage points, so the adjustment for response bias must be used. 


A B C D 
Employed 
__niverse  ___Completed _____ Employed ____ Estimated (A*C)/B 
Employed 300 240 180 225 


Not Employed 80 _ 50 15 24 
Total 380 290 


Employment Rate = D/A*100 = 65.5 


5. 85 weeks. Ali participants are included in the calculation of average number of weeks worked, 
regardless of whether they worked during the follow-up period. Any numbers in the ratio of 15 to 85 would 
give you this answer. 
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Guide for Setting JTPA Title II-A and Title i!l (EDWAA) Performance Standards for PY 1990 U.S. Department 
Labor Employment and Training Administration, November 1990. 


The ERIC Clearinghouses have a wealth of useful information. Look into their resources. The Eric 
Clearinghouse on Adult, Career and Vocational Education; Center on Education and Training for 
es Cae enn Saw nen 1900 Kenny Rd., Columbus, OH 43210; (614) 292-4353; (800) 848- 
43) 


a Error and bias can alter the usefulness of your postprogram data. Learn what to look for 
and how to combat its effects. 


a Fully understanding the adjustment for non-response bias can be difficult. We will present 
some easy to understand examples. 


a We provide a checkiist to help you minimize and control for error and bias. 
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Published by the Employment and Training Administration, U.S. Department of Labor, Rm N5629, 200 Constitution Averue, N.W., 
Washington, D.C. 20210 - (202) 219-5782 


IN as te 
—~ MODULE 


Follow-up data can never provide exact measures of the postprogram employment and earnings of participants. 
This module looks at error and bias at both the individual and the aggregate levels. This module also considers 
causes and effects of error and bias and suggests strategies for dealing with them. 


UNDERSTANDING ERROR AND BIAS 


| COLLECTING DATA IN AN IMPERFECT FORLD: | 


Follow-up data are never an exact measure of 
postprogram performance. Even if States and SDAs 
could reach every person who terminated from their 
programs, occasional mistakes in what participants 
report, interviewers record, and data entry clerks 
copy would still occur. If sampling is used, the 
resulting employment and earnings are, by definition, 
estimates of the employment and earnings of all 
terminees. And, since SDAs cannot reach everyone, 
there is always a potential for non-response bias. 


It is important to minimize the error and bias in your 
data and to understand what remains. A good place 
to start is with the difference between error and bias, 
because they have different causes, different effects, 
and different remedies. 


ERROR 


In everyday language, the word ‘error is 
synonymous with “mistake.” In the technical sense, 
error is made up of random differences between 
actual and observed values (i.e. the difference 


between the participant being employed and the 


participant saying he is employed). Saying that these 
differences are “random” means that they are not the 


result of any systematic process. 


the archer intended to hit the bull’s-eye, the shots 
reflect a lot of error. 


An example of follow-up error would be an 


interviewer or key entry clerk who occasionally 
transposes digits in recording weekly income. 


Bias, on the other hand, is the result of systematic 
differences between actual and observed values. in 
the next picture, the arrows are clustered tightly in 
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the upper right-hand quadrant. The archer is precise 
-- each shot lands close to all the others. But again, if 


the archer intended to hit the buliseye, these shots 
are biased. 


if an interviewer was consis:ently reading a question 
wrong or recording an answer incorrectly, bias would 
occur in the data. Error and bias are not an either/or 
proposition. Survey data typically involves both, and 
both can arise at either the individual or the group 
level. 


| ERROR AND BIAS IN INDIVIDUAL RESPONSES 


Two types of error can occur at the individual level. 
They are 1) mistakes and 2) lack of precision. 


For example, a respondent may forget that during the 
reference week he worked only 32 hours, instead of 
the usual 40 (a mistake) or he may approximate his 
earnings when he is not sure of the exact amount 


(imprecise measurement). 


In theory, error “evens out in the end.” When we add 
everything up and calculate rates and averages, we 
expect the error in individual responses to balance 
out. This should happen automatically when the 
sample size is large enough. However, in small SDAs 
or within small groups of participants, error can 
cause significant, misleading results. Consider the 
effect that recording a participant's weekly earnings 
as $2 instead of $200, would have on average weekly 
earnings, especially in a small SDA. 


Attention should be paid to extreme or unusual 
values during both the interview and data entry 
processes. Interviewers need to make sure the 
terminee understood the question and that 


responses are Clear, with any explanations appended 
to survey data where necessary. 


Using a data entry process that “flags” unusually 
large or small numbers, and that allows a second 
person to “verify” the first entry will help identify and 
correct these errors. 


T.AG.: pp. 68-69. 
bi 


When the terminee creates the error, we have a 
different problem. If terminees consistently tend to 
round their reported wages up, the pattern of 
responses will be biased upward. Someone who 
earns $5.28 an hour and rounds up to $5.30 an hour 
will be introducing an error that can bias the 
information. Questionnaire design, pretesting and 
interviewer training are the places to prevent biased 
responses to individual questions. Such bias is 
Cifficult to detect and correct after the fact. 


Tae ~=MOD 1, pp. 5-6; 5, pp. 34-35, 38-40. 


Your systems for training staff, identifying potential 
errors, eliminating systematic biases, and ensuring 
high quality data are very important to your ultimate 
utilization of the data. T.A.G.* addresses the 
problems of data validation in Module 11. It is very 
important that your systems themselves be non- 
biased, with equal attention given to correcting all 
errors, not just selective ones. 


T.AG.: pp. 87-89. 
a 


While other sources of information may be used to 


identify potential errors in reporting, only the terminee 
can resolve differences. Even the unemployment 


insurance wage record system has mistakes in it, and 
employer records are not always accurate either. 


The time to address accuracy is during every single 
step of your process. Attention to accuracy should 
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be part of quality control, not only for the purpose of 
correcting individual records, but also for 
identification and correction of system weaknesses. 
*k & 


fini f [ ERROR AND BIAS IN GROUP DATA | fii t 


Although sampling reduces the cost of follow-up, it 
introduces an additional source of error. Sampling 


error is the difference between the data your sample 
gives you and what you would have found out if you 
had interviewed everybody. 


lf you drew several different samples from the same 
population and calculated average weekly earnings 
at follow-up, you would expect to get slightly different 
results each time. If the samples were all randomly 
selected, each would be an equally valid estimate of 


average weekly earnings in the population. 


The bigger the sample, the better the estimate is 
likely to be because the sample will be more 


representative of the population from which the 
sample was taken. 


When you do sampling, you end up with a measure 
called “confidence intervals.” The confidence interval 
describes 2 range of values within which the total 
population is likely to fall, with some specified degree 
of probability. 


The minimum sample sizes for JTPA follow-up have 
been set at the 95% level of confidence with a 
sampling error of plus or minus five percentage 
points. That means there is a 95% chance that an 
SDA's true employment rate at follow-up can be 
assumed to be within five percentage points of its 
sample estimate -- if the estimate is unbiased. 


States and SDAs can narrow this margin of sampling 
error by increasing their sample sizes above the 
required minimum levels. They can eliminate it 


altogether by interviewing everyone. 


T.AG.: pp. 23-27. 
a 


Sampling works due to the statistical rules of 
probability. it is not our intention to explain this in 
great detail, but rather to provide some basic 
principles that are involved in the sampling process. 


a Sampling error is based on the size of the 
sample in relation to the size of the 
population from which it is selected. 


A common misconception is that you need to have a 
certain percentage (10% or 20%) of the people in a 
sample. When the population is large enough 
(usually 50,000 or more), we call that an “infinite 
population.” A sample of 400 people will have the 
same sampling error if it is selected from a State of 
100,000 or of 1,000,000 residents. 


But if the population size drops down considerably, 
we now operate with “finite populations.” This is what 
the JTPA sampling rules are concerned with. 


JTPA samples at the SDA level, and the number of 
persons that SDAs terminate over a year can vary 


greatly. The JTPA sampling requirements try to keep 
all SDA data within the same confidence intervals. 


TAG.: p. 25. 
ee 


By following the minimum sample size requirements, 
an SDA with 140 planned terminees in a program 
year must attempt to interview 94% of them. An SDA 
with 340 terminees must attempt to interview 67%. 
And an SDA with 540 will attempt to interview 50%. 


Regardiess of the sampling percentage, each of 
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these three SDAs will have data with a confidence 
interval of plus or minus 5 percentage points. 


| To be a true, unbiased random sample, 


everyone in the population must have the 
same chance of being selected. 


TAG.: Appendix E. 
ee 


A sample's validity hinges upon the unbiased 
selection of participants. The 1990 TAG. clearly 
provides you with several alternative procedures to 
select your sample. 


Two statistical problems can occur which introduce 
bias and affect the quality of your data at the group 
level. The first of these is the non-response bias 
adjustment, dealing with the problem of response 
rate differences between terminees who were 
employed at termination and those who were not. 


TAG.: pp. 72-74. 
a 


The second of these is the actual survey response 
rate itself. The next two sections discuss these two 
major sources of bias. % % 


| UNDERSTANDING THE ADJUSTMENT FOR N 


ON-RESPONSE BIAS | 


USDOL requires that you employ the non-response 
bias adjustment in calculating your performance 
measures when the difference in response ‘ates 
between employed and not employed at termination 
is greater than 5 percentage points. Why? 


TAG.: Appendix G. 
eee 


Persons who were employed at termination will be 
easier to locate than persons who were not 
employed at termination. You may easily achieve an 
85% response rate for the former group while only 
interviewing 70% of the latter. 


Bias occurs when a factor which is related to whether 
or not a terminee is contacted is also related to the 
responses he or she would have given if successfully 
interviewed. The adjustment process which SDAs 
must use when the difference in the response rates 
between the two groups exceeds 5 percentage 
points is designed to correct this problem. 


The non-response bias adjustment is based on the 
assumption that postprogram outcomes are 
essentially the same for those who are interviewed 
and for those who are not contacted. 


The “Worksheet to Adjust Follow-Up Performance 
Measures for Non-response Bias" provides an easy- 


to-use framework for making the adjustments. The 
explanation which follows attempts to demystify the 
worksheet. 


The adjustment process uses a technique of 
weighted averaging. The JASR requires you to report 
an overall employment rate at follow-up for the SDA. 
This rate includes both persons employed at 
termination and persons not employed § at 
termination. 


Because persons employed at termination are 
usually easier to locate and interview, an average 
employment rate will often over-represent these 
people. The following table provides an example of 
how the non-response bias adjustment works and 
what its impact is. 


The non-response bias adjustment assumes that the 
people you did not interview had the same 


experiences as the people you did locate and 
interview. 


Line 1 -~ The SDA terminated 400 people, 75% 
(300/400) of whom were employed at termination. 


Line 2 -- In all, 275 people were interviewed, 82% of 
whom were employed at termination (225/275). 
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Line 3 — The response rate overall was 69%, but it 
was 75% for employed at termination and only 50% 
for not employed at termination. Since the two group 
response rates (columns A and B) differ by more than 
5 points, the non-response bias adjustment must be 
used. 


TABLE 2: Adjustment for Non-response Bias: An Example 


Line 4 — Of the 275 people, the follow-up survey 
found 190 (69.1%) to be employed at follow-up. This 
is the unweighted average. 


" Not Nt 
EMPLOYED EMPLOYED 

1. Number Terminated 300 100 400 
2. Number Interviewed 225 50 275 
3. Response Rate 75% 50% 69% 
4. Number Employed Among Those Contacted 180 10 190 
5. Unadjusted Employment Rate at Follow-up 80.0% 20.0% 69.1% 
6. Estimated Number of Terminees Employed at Follow-up 240 20 260 
7. Adjusted Employment Rate at Follow-up 80.0% 20.0% 65.0% 


Now here’s the problem: 


Because we only reached 50% of the people who 
were not employed at termination, these people are 
under-represented in our average. The nonresponse 
bias adjustment requires us to “guess” what the 
follow-up employment rate for everyone would have 
been. To make this ‘guess’, we multiply the 
employment rate in line 5 times the number 
terminated in line 1 and put the answer in line 6. 


Line 6 -- This is the guess and it says: “If we had 
interviewed all 300 persons employed at termination, 
we guess that we would have found 240 of them still 
employed at follow-up. if we had interviewed all 100 
persons not employed at termination, we guess that 
we would have found 20 of them employed at follow- 
up.” 


What has happened now is that the total estimate of 
260 persons employed at follow-up is based on 
numbers that wash out the differences caused by the 
two different response rates. 


Line 7 -- The important column is C because that is 
the number that goes on the JASR. The new 


weighted average for the employment rate at follow- 
up drops to 65.0% (260 / 400). 


The 65.0% is a much more accurate guess of the true 
employment rate than is the 69.1%. 


“Not fair” is a common response to this calculation. 
But this process is actually very fair. Since persons 
employed at termination only constituie 75% of your 
terminees, their experiences at follow-up should only 
constitute 75% of your rate. 


The current procedure may not look fair to some, but 
it does give every SDA the benefit of the doubt as to 
the follow-up experiences of terminees who could not 
be located. 


This process looks complicated but the worksheet in 
the 1990 T.AG. is really very simple to complete. 
Practicing it will help you understand where the final 
reporting numbers come from and it will reduce the 
number of surprises you may have at the end of the 


year 
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CAUTION. States and SDAs which 
maintain follow-up data in a file which is 
seperate from participant termination 
data may be incorrectly applying the 
adjustment for non-response bias. 


Tha previous example assumes the SDA 
attempts interviews with everyone and 
does not sample. Line 1 is both the 
number of people eligible for follow-up 
and the sample selected. 


The weighting procedure must be based 
on the true number of people eligible for 
follow-up. This number could be 
different from line 1 if the SDA had late 
terminations. At the time the JASR 


calculations are made in August, i is important that 
MiS verify the true number of terminations in the 


participant data base so that the adjustment 
calculations can be made. 


The adjustment procedure, once again, is based on 
the assumption that the people interviewed and not 
interviewed are identical. That is a pretty broad 
assumption. For your program planning purposes, 
the overall response rates are very important and if 
low, can be giving you some inaccurate information. 
That comes next. 3 3 


| COMPLETION RATE BIAS 


Discussions about statistical corrections, confidence 
intervals and error are assuming that error is random. 


Using follow-up data in program planning assumes 
that you can trust it, within specified statistical limits 
(Le., plus or minus 5 percentage points). You can 
make that assumption reasonably when you have 
100% response rates, but very few SDAs have that 
success. 


When your response rate is less than 100%, you are 
now introducing non-random error into your data. 
Statistically correcting for this kind of error in most 
social research data sets is not possible. But the 
JTPA data do provide the information necessary for 
such adjustments, although they are admittedly very 
complex and beyond the technical capability of most 
SDAs and many States. 


Take the following two examples reported to us by 
one State. 


ONE. Response rates by age groups were 78% for 
people 22-29; 81% for people 30-39; and 83% for 
people 40-49 and 91% for those 50 and older. The 
follow-up data count the responses for older 


participants more than for younger because of the 
response rate differences. 


TWO. Response rates by educational attainment 
were 74% for those not completing high school, 83% 
for those with diploma or GED, and 85% for those 
with post-secondary education. The follow-up data 
count the responses for better educated participants 
more than for less educated because of the response 
rate differences. 


Many pages of such examples could be presented. 
Response rates vary by gender, by race, and by 
other important factors. Statistically balancing them 
all is extremely difficult. The best answer, in the 
reach of any State or SDA, is taking the steps to 
ensure that survey response rates are as high as 
possible 


The impact on the data, for instance, is more of a 
concern to an SDA with a 65% overall response rate 
than to the SDA with a 90% response rate. The lower 
your response rate, the less you can trust that your 
-. reflect your SDA’s performance. 
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ERROR, BIAS, AND PERFORMANCE STANDARDS 


States need to be mindful of the fact that follow-up 
data are subject to sampling error and non-response 


obtained can be used for vendor and service 
provider evaluation. 


es Average weekly earnings are not measured 
precisely enough to merit use of fractions of 
dollars. Consider giving SDAs the benefit of 


the doubt by rounding average weekly 
earnings up to the nearest full dollar. 


& Use the adjustment for non-response bias 
consistently in ali cases. if the response 
rates :or participants are exactly the same, 
whether they were placed or not, the 
adjustment will have no effect. if response 
rates Ciffer, t will fine-tune your estimates of 


actual performance. 


if sampling is being used in some SDAs, 
consider using the tolerance intervals 
provided with the performance standard 
worksheets — # you are not using them 
already — to accommodate sampling error. 
Accept any value within the tolerance interval 
as meeting the standard. 


Look at SDA questions and concerns about 
the accuracy of individual participant 
responses from a system perspective. Use 
what you learn to improve the system. 


Be prepared to work with SDAs on tracking 
down possible sources of bias and 
determining whether they are having adverse 
effects on performance. if problems with 
bias occur and are not addressed adequately 
in the adjustment process, consider making 
Governors’ adjustments as needed. 


Make sure that staff responsible for 
performance standards have a good working 
knowledge of error and bias and how they 
may effect observed performance. Support 
them in providing training and technical 
assistance to SDAs on technical aspects of 
the postprogram standards. 3 3 


'% | MINIMIZING ERROR AND BIAS: A CHECKLIST | 


pe | MOD 1, pp. 4-7. 
Ce 


_____ Survey questions are clearly stated and easy 
to understand. The survey flows in a logical 
way and supports accurate collection of 
postprogram data. 


____New survey questions or procedures are pre- 
tested before they are used. 


po | MOD 4, pp. 25-26 
a 


Participants are aware of follow-up, its 
importance, and that they may be contacted. 


pv MOD 2, pp. 9-11. 
eee 


if sampling is used, the costs and benefits 
associated with that choice heave been 


b5 


random fashion. 


MOD 3, pp. 17-18, 21-23; 4, pp. 27-31. 


_._ A variety of checks are used to ensure that 
data entry errors are found and corrected. 
oK 
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States and SDAs which maintain follow-up data in a 
file which is separate from participant termination 
data may be incorrectly applying the adjustment for 
non-response bias. 


This adjustment should be made based on all 
terminees. Where late entry of terminations is a 
problem, some terminees will be missed # the 
adjustment of non-response bias is made based on 
the terminees identified for follow-up. Because status 
may be related to late terminations, they are likely to 
be a source of bias. Fortunately, correct use of the 
adjustment for non-response bias should take care of 
this problem where the number of terminations 
missed is small. 


a We explain the use of postprogram data in performance management. 


Ss Postprogram support service can make a difference in your workforce retention rates. 


a There are many postprogram indicators that will provide you with feedback on local 
programs. Learn more about how to use these. 


| Predicting postprogram performance is not as difficult as t may seem. We will show you 
how to combine termination data with follow-up data in a creative manner. 
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In this Module of T.A.G.* we look at postprogram performance standards and performance management. Our 
topics span the participant, project, and SDA levels. They include postprogram support services, useful 
postprogram indicators of program and project performance, and predicting postprogram performance from 
termination outcomes. 


POSTPROGRAM DATA AND PERFORMANCE MANAGEMENT 


Effective performance management relies heavily on 
the creation of feedback mechanisms which allow 
decision makers to observe the effects of their 
actions over time. 


PARTICIPANT 
‘OUTCOMES iz 
POSTPROGRAM 
ACTION DAT, 
¢ MANAGEMENT 
TION 


FEEDBACK LOOP FOR PERFORMANCE 
MANAGEMENT 


in the first years of JTPA, SDAs assessed thelr 
program through termination-based performance 
measures. The shift to follow-up meesures has made 
performance management more challenging. 
Postprogram data are less timely and the technical 
complexity of the data collection process makes 
them more difficult to interpret and use. 


On the other hand, postprogram performance 
standards focus attention on job quality and long- 
term employability. In doing so, they redefine the job 
training system's commitment to is adult 
participants. This module explores performance 
management within the context of that new 
commitment 3 3 


POSTPROGRAM SUPPORT 


While some SDAs and PICs have expressed concern 
that postprogram standards give them less control 


over thelr outcomes, these requirements merely 


require a revision of strategy. SDAs are increasingly 


A TAG.: pp. 


information in the 1990 TAG. Is ‘kheyed” by this symbol. 


_ 
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recognizing the importance of both program and 
postprogram support services as key pieces of the 
job training puzzle. 


IMPORTANCE OF SUPPORT SERVICES 


if all JTPA had to do was to provide job training, this 
would be a relatively easy job. The challenges to 
JTPA are to provide services within a constantly 
changing economy fer persons facing an infinite 
diversity of individual situations, problems and 
oxperiences. 


Successful outcomes in many instances have 
resulted from judicious and appropriate use of JTPA 
supportive services during program participation. 


JTPA also permits SDAs to provide support services 
to participants for up to six months after they leave 
the program. This provision reflects an 
understanding on the part of Congress that entering 
employment does not ensure successful continued 
employment. The early weeks on the job, in 
particular, are cruciai. 


During those early weeks, new workers must come to 
terms with the culture and expectations of the 
organization. They find themselves in a period of 
testing, during which their employers view their 
performance as a predictor of their future worth. At 
the same time, the job may introduce new stresses 
into the worker's personal and family life. Even the 
best-trained participants may neeji moral support 
and practical assistance during this period. 


Postprogram standards focus attention on this critical 
period because it is retention which makes or breaks 
employment at follow-up for SDAs, once they have 
achieved adequate entered employment rates. 


For example, suppose an SDA has an entered 
employment rate of 75%. The survey finds that of the 
25% not employed at termination, 25% have jobs at 
13 weeks. The “not employed at termination”, then, 
will add 6 percentage points (25% times 25%) to the 
overall employment rate at follow-up. 


The remaining portion of the SDA’s employment rate 
at follow-up depends on the SDA’s_ work- 


force retention rate -- the percentage of those who 


Mao 100 


follow-up. 
employment rate at follow-up would be, for workforce 
retention rates ranging from 90% down to 50%, 
including the 6 percentage points from the ‘not 
employed at termination” group. 


Retention Rate | dow Up 
90% 74% 
80% 66% 
70% 59% 
60% 51% . 
59% 44% 


Workforce retention does make a difference. 


A variety of postprogram support services can be 
provided to participants who enter employment. 
Certainly case management services, with regular 
contact and referral to specialized assistance as 
needed, can be considered. Support groups are 
another strategy which may be effective, especially 
when they build on patterns of mutual support 
established during training. 


Services such as_ child care, transportation 
assistance, and health care can be continued for up 
to six months after termination. Even if the SDA does 
not fund these services, it can act as a broker to 
ensure that such basic needs are met. In addition, 
SDAs can support participants by working with them 
and their employers to resolve problems. Such 
services may be especially important for participants 
with special barriers to employment such as physical 
handicaps or limited English-speaking ability. 


The basic message underlying postprogram 
performance standards is that an SDA’s commitment 
to its participants need not end upon entering 
employment. This should simply mark the transition 
to a new phase of service. 


By studying the patterns of workforce retention 
among different subgroups of terminees, an SDA 
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| _POSTPROGAM INDICATORS FOR PROGRAM MANAGEMENT | 


Employment and earnings are not the only, or even 
the most informative, postprogram indicators which 
can be used in managing local programs. The 
measures in this section go beneath the surface of 
postprogram performance. They look at aspects of 
performance which have direct relevance to program 
management, but are not necessarily incorporated 
into the formal reporting requirements of mandatory 
questions. 


TAG.: pp. 13-16. 
eee 


Because program management focuses heavily on 
activity at the project level, these measures are most 
useful in SDAs which do not sample. Depending on 
the sampling percentage used, the measures still 
have value at the SDA level, but may be much less 
useful for project or contractor assessment. 


WORKFORCE RETENTION 


Perhaps the most useful postprogram indicator is the 
employment rate at follow-up for participants who 
were employed at termination. The ‘workforce 
retention rate,” is an important indicator of program 
quality. A workforce retention rate below 80% 
suggests that participants are entering the workforce 
but are unwilling or unable to keep ongoing 
employment. Together, entered employment rate 
and workforce retention rate measure the continuity 
and quality of service being provided. Both of these 
can be useful measures in establishing performance 
expectations for contractors. 


RETENTION RATE IN TERMINATION JOB 


No one in the JTPA system really expects that the 
first job post termination will be the participant's last 
job. Sometimes these are the only jobs available, are 
good entrytevel experience, or are good 
opportunities. Routinely examining this rate by 
service activity, employer and by service contractor, 


should be done to make certain there are no 
particular systematic biases. 


CAUTION: Measuring retention in the 
termination job by computer requires the 
name of the termination employer to 
exactly match the name collected at 
follow-up. Acme, inc. and ACME INC will 
be considered two different companies 
by the computer in a simple match. If a 
termination company is a_ holding 
company, or does business under a 
different name, you cannot match 
employers unless the interviewers 
carefully explore this and record the 
match property. 


UNVERIFIED EMPLOYMENT AT TERMINATION 


Any person who is recorded in your system as “not 
employed at termination” but who reports at follow- 
up that they worked all 13 weeks has unverified 


employment. 


This is an indicator that few people look at. if the 
system is working properly, the rate should be 0.0% 
That would mean employment on all participants wis 
correctly recorded at termination. But if this rate is 
not zero, then some persons who were employed at 
termination were missed in the counting. 


Most follow-up surveys are not designed to capture 
this rate fully. All follow-up programs are capable of 
capturing it partially. To fully capture unverified 
employment at termination, you would have to add a 
question to your survey which specifically asks the 
person if they were working during the week in which 
they were terminated. 


U | 


po | MOD 8, pp. 58, 59. 
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Examine these cases carefully. As Module 8 
discussed, random error will always get into your 
data. You are most concerned with any systematic 
bias. 


SELF-PLACEMENT RATE 


The employment rate at follow-up for participants 
who were not employed at termination is the “self- 
placement rate.” This measures the extent to which 
people who were not employed at termination, 


subsequently got jobs on their own. 


As a performance management indicator, this rate 
potentially raises many questions. Did the provision 
of postprogram support services have any impact on 
the rate? Is there any pattern by service provider? 
Are providers and SDAs failing to maintain contact 
with participants? Should any of these people really 
have been able to have employment at termination? 
Has the rate changed over time? Do these rates vary 
among SDAs? 


FULL-TIME EMPLOYMENT RATE 

Labor economists are increasingly concerned that 
the American economy will shift to more part-time 
employment through the balance of this decade. The 
financial impact to business will be lower employee 
benefit costs. The financial impact to employees will 
be the necessity of holding more than one job to 
maintain total earnings. 


Tracking your full-time employment rate across time 
will tel! you something about how the local economy 


is changing. 


CAUTION: For this measure to have 
validity (i.e., for it to mean what you think 
it means), you have to be recording 
hours worked for each job worked. For 
most terminees that is no problern since 
they only have one job in the reference 
week. If you lump all hours from all jobs 
together for people working two or more 
jobs, then this measure cannot be used. 


The number of respondents who entered 
employment at termination and who were 
employed at follow-up, divided by the number of 
respondents who entered employment at 
termination. 


RETENTION RATE IN TERMINATION JOB 


The number of respondents who were still 
employed in their termination job at follow-up, 
divided by the number of respondents who were 
placed and whose placement job was verified 
during the follow-up interview. 


The number of respondents who were not 
recorded as being employed at termination but 
who reported working all 13 weeks, divided by the 
number of respondents who were recorded as not 
employed at termination. 


SELF-PLACEMENT RATE 


The number of respondents who were not 
employed at termination but who were employed 
at follow-up, divided by the number of 
respondents who were not employed at 


termination. 


FULL-TIME EMPLOYMENT RATE 

The number of respondents who were employed 
at follow-up and who worked more than 30 hours 
during the 13th postprogram week, divided by the 
number of respondents who were employed at 
follow-up. (Sorne people prefer to define full-time 
as 32 hours a week. Use what is appropriate for 
you.) 


LABOR FORCE NON-PARTICIPATION RATE 
The number of respondents who were neither 
employed nor looking for work at follow-up, 
divided by the number of respondents 
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LABOR FORCE NON-PARTICIPATION RATE 


Another program management issue which you may 
want to address in your follow-up survey is labor 
force participation. One State which has collected 
postprogram data on labor force participation for a 
number of years has found that about 15% of those 
contacted were neither employed nor looking for 
work at follow-up. 


The reasons why participants may be out of the labor 
force include iliness, pregnancy, return to school, 
and the need to care for family members. Measuring 
labor force participation gives you an indication on 
the upper limits on your employment rate at follow- 
up. Unusually low labor force participation rates can 
point to service providers who may be enrolling 
participants who are unsuitable candidates for job 
training because they are unwilling or unable to work. 
Or it may point to service proviciers who are failing to 
address their participants’ barriers to workforce 


participation. 


To collect the data to calculate this indicator, you 
would simply need to ask persons not employed at 
follow-up what they were presently doing. 


OTHER POSTPROGRAM INDICATORS 


Creativity, initiative and individual needs are the 
driving motivation for collecting and analyzing other 
indicators. This section has mainly focused on what 
can be learned from data everyone routinely collects. 


You may want to consider adding one or more 
questions to determine terminee satisfaction with 
services. Most participants will tend to report that 
they are satisfied with the services unless they were 


Other questions which may be added to generate 
new indicators might include welfare dependency at 
follow-up, availability of health and other benefits, 
and other issues. 


There is literally no end to the number of indicators 
you can collect which could be useful to your 
program. Networking with other States and SDAs is 
a good source of ideas. This is an area where 
professional assistance in survey design may be 
helpful, even if you retain responsibility to collect the 
data. *%* 


| PREDICTING POSTPROGRAM PERFORMANCE 
FROM TERMINATION OUTCOMES 


In the past, performance management was closely 
linked to termination performance standards. When 
postprogram standards were introduced, SDAs 
reported difficulty in determining whether their adult 
programs were on target relative to standards. 


Postprogram data are not available until 3 or 4 
months after the participants contacted have left the 
program, but SDAs need not wait until the results of 
13 week foliow-up are received before they begin to 
assess their performance. 


One strategy for dealing with this situation is to 
predict postprogram performance from termination 
outcomes combined with the SDA's past 
performance. Doing so will enable an SDA to: 


a Get an early reading on its performance 
relative to standards. 


a Keep track of changes in performance, and 
especially workforce retention, over time. 


a Continually refine the prediction process 
based on experience. 


EY 
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PREDICTING FOLLOW-UP EMPLOYMENT RATE 


The formula for predicting employment rate at follow- 
up is: 


E = (Tx Rp) + (/1-1) x Rp) 


Where: 

E = Employment Rate at Follow-Up 

T = Employment Rate at Termination 

Rp = Expected Employment Rate at 
Foliow-Up for Employed at Termination 

Ry = Expected Employment Rate at Foliow- 
up for Not Employed at Termination 


Rp can also be called an expected retention rate, 
although in this case, retention is ‘workforce 
retention” rather than retention in the same job. 


TAG.: p. 75. 
erent 


SDAs have relatively little control over what happens 
to the participants not employed at termination once 
they leave the program. However, they can have an 
effect on both employment and retention rates. 
Dividing employment at follow-up into its component 
parts makes the contributions of termination 


employment and employment retention ciear. 
Here's an example: 


Suppose an SDA has terminated 400 participants, of 
whom 300 were employed at termination. 
Experience has shown the SDA staff that about 80% 
of these people will be employed at follow-up, while 
about 24% of those that are not employed at 
termination will be employed at follow-up. 


Using the above equation, the SDA predicts that its 
employment rate at follow-up will be 66%. It expecis 
that 60% of its terminees will be employed and 
retained and another 6% will get jobs during the 


follow-up period. 
E = (75.0% x 80.0%) + (25.0% x 24.0%) 


E = 60.0% + 6.0% = 66.0% 


After follow-up has been completed, the SDA repeats 
this calculation using actual employment rates at 
follow-up. it finds that its retention rate was only 
70%, so that only 525% of its terminees were 
employed and retained, but that 25% of those not 
employed at termination did find jobs. 


E = (75.0% x 70.0%) + (25.0% x 25.0%) 
E = §25% + 6.3% = 58.8% 


The SDA can now look at retention by each project to 
identify the source of its problem. 


PREDICTING FOLLOW-UP EARNINGS 


Average weekly earnings can be predicted based on 
experience, too. A conversion index is calculated by 
dividing average weekly earnings for a baseline 
period - such as the previous program year -- by 
average wage at termination for the same period. 
The SDA's current average wage at placement is then 
multiplied times this index to predict average weekly 
earnings at follow-up. The size of this index will be 
determined largely by the extent to which those 


placed were employed full versus part time. 


The formula for predicting weekly earnings at follow- 
up is: 


C = Wo/Ho 
WwW, =CxHy, 


Where: 


year 
Ho Hourly Wage at Termination, previous 
program year 
Cc = Conversion index 
Wy Projected Weekly Earnings, current 
follow-up year 
Hy Hourly Wage at Termination, 
current follow-up year 
For example: 


Last year, an SDA had an average wage at placement 
of $5.50 and average weekly earnings at follow-up of 
$220. Therefore, its conversion index is 40. 
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C = $220 / $5.50 = 40 
At the end of the fourth quarter, its average wage at 
placement is $5.65. it predicts that its average 
weekly earnings at follow-up for these terminees will 
be $226. This figure is comfortably above the SDA’s 
estimated standard. 


W, = 40x $5.65 = $226 


After follow-up is completed, the SDA finds that its 
average weekly earnings for its fourth quarter 
terminees was $224. This is close so the SDA 
continues to use 40 as its conversion index. 


Had its average weekly earnings been significantly 
lower than expected, i would have looked at the 
extent to which participants were working part time 
rather than full time and sought ways to promote 
placement into full time employment. 


CAUTION: Terminology can be very 
important. When calculating these rates, 
t is critical that you keep in mind what 
time periods you are addressing. 


T.AG.: pp. 77-78. 
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PROGRAM YEAR -- This term refers 
specifically to the program planning 
year followed by States and SDAs. 
This is always July through June. 


REPORTING YEAR -- This is the 
annual period used for the follow-up 
survey reported on the JASR and the 


WAPR. This is always April through 
March. 


The follow-up performance mea- 
sures are based on Reporting Year 
data, while the balance of the JASR 
uses Program Year data. The first 
quarter of a postprogram reporting 
year is the fourth quarter of your 
previous program year. Be sure you 
are specific in the time periods you 
use. 


CALCULATE USING BOTH YEARS 


JTPA accor'ntability mechanisms are based on the 
follow-up reporting year. Predicting these measures 
is extremely important to States and SDAs. Module 9 
has focused aimost exciusively on this type of 
program management. 


Do not, however, forget that you began the program 
year with a plan. if you are using the performance 
indicators discussed in this module, you should also 
calculate those using the program year time period 
as well as the reporting year period. 


if you assess the quality of your plan with reporting 
year estimates and data, then you have a one quarter 
mis-match between your indicators and your plan. 


While the differences may appear to be slight, in the 


recently? Know your numbers. % 3 


SB a 


In management information systems which rely 
heavily on use of “canned" report formats, MIS staff 
can support performance management by making 
project-evel follow-up reports available. 


These reports should allow the user flexibility with 
regard to selection of the termination dates covered, 
and they shoul? display summary information by 
plecement grou,s as well as for all participants. It is 
important that the number of follow-up interviews 
completed and the project's response rate be 
included to help users get a sense of how confident 


they can be in the representativeness of the 
outcomes shown. 


employer at termination and the employer at follow- 
up, as well as employment status, number of hours 


case by case outcome review. % ok 
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Although not a full list, several national organizations are very active in employment and training research. 
You might want to check what they are doing and publications they may have available. 


National Center for Research in Vocational Education; 1995 University Avenue, Sulte 375, Berkeley, CA 
94704; (415) 642-4004. 


SRi International, Employment and Training Research Program; 335 Ravenswood Avenue, Menio Park, CA 
94025; (915) 326-6200. 


Manpower Demonstration Research Corporation; Three Park Avenue, New York, New York 10016; (212) 
532-3200. 


AST Associates; 55 Wheeler Street, Cambridge, MA 02138; (617) 492-7100. 
James Bell and Associates; 3062 N. Quincy St., Arlington, VA 22207; (703) 528-3230. 


Appropriate Solutions, Inc.; 1357 W. Lane Avenue, Suite 207, Columbus, OH 43221; (614) 486-5295. 


a in Module 10 we will begin to expiore Level 3 utilization of follow-up data. 
a Postprogram data can become a valuable asset in your program planning and evaluation. 
a Learn about different ways in which o organize your data and what it can tell you. 


1] What type of computer software do you need in order to transform your data into reliable 
information. 


YU 
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In this issue we look at Level 3 Utilization -- use of follow-up for program evaluation, planning, and policy analysis. 
While not all SDAs have the technical resources to do complex analysis of postprogram data, virtually all can ask 
and answer meaningful questions about program outcomes if they have the capacity to generate simple tables and 
crosstabulations. 


Ey [Lever 3: THE PAST INTO THE FUTURE | map 


Different dictionaries carry different definitions of the 
verb “evaluate.” They do agree that the action taken 
is one of determining worth and of doing appraisal. 


T.A.G.* has been devoted to many different issues 
and topics which affect the quality of your evaluation 
system. Although we have talked about utilization in 
terms of three different levels, we realize that the 
dividing lines between these three levels are not 
always Clear. 


Many pages have been devoted to Level 1 -- fully 
meeting the Federal reporting requirements related to 
performance standards. This constitutes the most 
basic use of the follow-up data. 


Level 2 moves into actively using the data for 
performance management. Most of these activities 
focus on past behavior. Module 9 presents ideas on 
indicators to use for examining what has happened. 


At Level 3, the past is relevant to the extent that it 
helps change the future. This is program evaluation 
and policy analysis. You can find yourself doing 


some of the same things in Level 3 as you do in Level 
2. The key difference is the: context. 


Many Level 2 activities can be performed in a value- 
free context. Level 3 cannot. At Level 3 you have 
specific program objectives to assess. You have 
specified the criteria in advance that you will use for 
your appraisal. You have indicated your intended 
program impacts. Your data needs are clear, formats 
ere defined, and procedures to collect and process 
data are in place. 


PROGRAM OBJECTIVES 


USDOL provides you with the performance 
standards. Most social programs -- local to federal -- 
lack such a standards system. 


Additional flexibility and responsiveness is available 
through the Governor's adjustment. This helps each 
State to establish fair and equitable standards 
reflecting the challenges faced by their different SDAs 
which the national model cannot anticipate. 
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For those truly working at Level 3, the performirws 
standards are but a departure point. They dc i 
preciude States and SDAs from establishing more 
expansive evaluation objectives on their own. 


To work at Level 3, you have thoroughly studied your 
system, identified the barriers to success, and know 
how to tailor your program to meet your specific 
goals and objectives. 


You have a program objective that you want to 
evaluate. How are you going to measure that 
objective? You enter the realm of “operationalizing” 
your objectives. What questions are you going to 
ask, what data are you going to collect, and how will 
you rate your level of success toward meeting your 
objectives? 


More than just deciding on your assessment data, 
you should also be establishing your thresholds. For 
instance, suppose that your SDA wants to have a 
“quality of service” indicator. You further decided 
that this will be measured by including a question on 
the follow-up survey which asks people to rate, on a 
scale of 1 (low) to 10 (high), the treatment they feel 
they received in the SDA office. 


Based on the quality of service you wish to provide, 
and in discussion with your staff, you set the goal at 
70% of your participants giving scores of 8, 9 or 10. 


The reason it is important to establish goals is that 
when the actual data do come to you, you want to 


achieve many other objectives along the way. You 
may establish other benchmarks which different 
programs should achieve for different participants. 
Certain levels of resume preparation skiis, reading 
levels, math capability, and self-esteem goals are 
arnong the many other objectives that can be 
realized. 


Performance standards only deal with some of the 
key bottom lines of your program. in reality, you will 
have many important benchmarks along the way 
from application to termination. Level 3 assumes that 
you are assessing the pieces of your program as well 
as its overall impact. 


DaTA COLLECTION PLAN 


What pieces of data are being collected at what times 
and by whom? Knowing where your evaluation data 
is coming from is a very important part of your total 


assessment plan 
poe | MOD 5, pp. 41-42. 
ee 


You may modify intake forms, change questions on 
your follow-up survey, include a brief tennination 
interview, or add variables to your MIS. As you learn 
from your information, you will continue to improve 
your original data collection. 


Your data collection plan supports your overall goal 
of producing a detailed, high quality, and timely 
evaluation of your program. 


You understand the people you are trying to serve. 
You know what the employers in your area are 
looking for and their plans for the future. You've 
carefuily assessed your past performance and have 
an agenda for improving your programs. 


Level 3 gives you control over your own future. ae 
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distinguishes Level 3 use of postprogram data is not 
the measures i uses but the analysis process itself. 
it is an exploratory process, driven by specific 
questions about the dynamics of service delivery. Its 
goals are to discover patterns which help explain 
what is going on beneath the surface of the program. 


Analysis relies on two very different types of 
knowledge: 


@ An understanding of the programs and factors 
which effect participant outcomes, and 


@ The technical know-how to accurately 
manipulate and display participant data. 


These two types of knowledge need not reside in the 
same individual. The analysis process can be a team 
effort, in which staff with computer skills support 
those with program knowledge. However, if either 
skill is missing, the process becomes problematic. 


Technical skill and program knowledge come 
together in the design of data formats which shed 
light on specific, planning-related questions. These 
formats allow patterns in the data to show through. 
initially, these questions may reflect basic program 
issues Or Current p dicy concerns. However, as the 
process of analysis continues, additional questions 
will emerge from the data. 


There is an art to asking questions of your data. It 
involves translating program concerns into a series of 
informal hypotheses, thinking about the patterns you 
would see in the data if your hypotheses were true, 
and organizing your data in ways which clearly reveal 
those patterns if they are present. 


The scenario which follows Wustrates the analysis 
process: 


You have a general concern about maximizing the 
workforce retention rate of participants--persons 
employed at termination who are also employed at 
13 weeks. Your hypothesis asserts that retention is 
related to job quality. Specifically, you believe that 
participants are more likely to stay in “better” jobs. 
The only measure of job quality you have for 
termination jobs is termination wage. But # your 


hypothesis is true, you would expect participants with 
higher termination \ages to have higher workforce 
retention rates. 


To test your hypothesis, you create a table in which 
participants who were employed at termination are 
grouped on the basis of their placement wages. You 
calculate the workforce retention rate for each group 


and find that, in fact, workforce retention rates 
increase as wage levels increase. 


TABLE 1 
Workforce Retention by Placement Wage 


Workforce 


Placement Wage Retention Rate* 


$4.00 - $4.99 60% 


$5.00 - $5.99 65% 


$6.00 - $6.99 78% 


$7.00 - $7.99 33% 


$8.00 and higher 90% 


*See Module 9, pg. 68 for definition of “Workforce 
Retention Rate” and how to calculate it. 


This finding ieads to additional questions. For 
example, does gender make a difference? You 
prepare a table of workforce retention rates by 


placement wage groups to see if the relationship 
between placement wage and workforce retention is 


the same for both men and women. 
TABLE 2 
Workforce Retention by Placement Wage 
wees Workforce Retention Rate 
Men Women 
$4.00 - $4.99 50% 64% 
$5.00 - $5.99 60% 68% 
$6.00 - $6.00 80% 76% 
$7.00 - $7.99 85% 75% 
$8.00 and higher 92% 82% 
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You discover that, generally, the pattern is the same 
for both groups, but f appears to be much stronger 
for men. You suspect that different factors influence 
workforce retention for men and women. As you 
continue your analysis, you look at men and women 
separately. You keep your analysis focused on 
issues over which you have some degree of control, 
such as training activities, target occupations, service 
delivery models, and supportive services. As your 
understanding grows, you discuss your findings — 
and the story they tell — with your colleagues. 


Together, you consider the implications of what you 
have learned for your program design. % 


Analysis does not provide ‘the answer," and 
it certainly does not tell you what to do with 
what you learn. But it does make the inner 


workings of your program clearer. And, ovar 
time, it allows you to observe how program 
outcomes change as your program design 


REVEALING PATTERNS: 
APPROACHES TO ORGANIZING DATA 


A broad array of questions can be addressed using a 
few basic methods of organizing data. 


FREQUENCY DISTRIBUTIONS ANC OTHER ONE-Way 
TABLES 


Routine management reports rely heavily on 
summary statistics such as counts, rates, and 
averages. Those statistics often hide more than they 
reveal. Frequency distributions and other one-way 
tables reveal what summary statistics tend to hide. 


Frequency distributions show the number or 
proportion of participants in each of several groups. 
The groups define a single concept, such as 
educational status or number of hours worked 
(grouped by categories) in the 13th week after 


seeedsiaeisi 
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A frequency distribution is most useful when the 
variable has a limited number of categories. Numeric 
variables, such as age or income, should be re- 


coded into groups before being displayed. 


Numeric variables that are not re-coded can be 
analyzed by looking at averages (or means) and 
medians. A mean is the numerical average of a 
variable. Add up all the values and divide by the 
number of values. A median is the midpoint of your 
variable. if you have 201 cases the median is the 
value (lowest to highest) of case 101. If you have 200 
cases, the median is the average of the value of 
cases 100 and 101. 


TABLE 3 
Average Income ai Follow-Up 


1385 


$262 


CROSSTABULATIONS AND OTHER TWO-WAY TABLES 


A crosstabulation displays counts or percentages, 
like a frequency distribution, but for two or more 
variables at the same time, such as placement status 
and gender or educational atiainment§ and 
employment at follow-up. All variables in a 
crosstabulation should have a limited number of 
codes. 


Ui 
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Employment at Follow-Up by Termination Status | 

Employment Status at “ollow-Up | 
Status Employed Not Employed Total 

Employed at Termination 60 300 

Not Employed at Termination 75 100 

Total 135 400 


consider the crosstabulation of termination status by 
employmert status at follow-up as shown in Tabie 4. 


When the data are displayed in this way, it becomes 
clear where the SDA's employment rate at follow-up 
(265/400, or 66%) is coming from. When one of the 
two variables is numeric, means are calculated for 
each group in the non-numeric variable. 


TABLE 5 __ 
Weekly Earnings by Gender 
Cases Mean 
Overall 1386 $282 
Female 655 $237 
730 $322 
correlation 


follow-up. This relationship raises questions 
regarding ways to overcome the gender gap as well 
as how to meet the educational needs of dropouts. 


Analysis is really a process of structured, systematic 
inquiry. It is a journey of discovery, not a destination. 
By asking your data to answer questions, you learn 
to ask better questions. 


Entire books and courses are devoted to statistical 
processing of data and the number of available 
analytic techniques go into dozens. Start with the 
simple techniques. They are the easiest to 
understand and can nonetheless be very powerful. 


While every technique discussed in this section can 
be accomplished with a hand calculator and 
patience, computers are highty recommended. % 


TABLE 6 


Average Weekly Earnings at Follow-up by 
Educational Status and Gender 


Women 
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ANALYSIS SOFTWARE: THE RIGHT TOOL FOR THE JOB 


Personal computers make low-cost data analysis 
possible for anyone who can get access to a data 
fie. There are many different products on the 
market. The right product for your organization 
depends on a variety of factors. There are several 
families of products which you may want to think 
about in deciding how to increase your data analysis 
capabilities 


Spreadsheet programs are great for displaying and 
manipulating summary data. Spreadsheets can 
import data from a variety of formats. They are 


Data base management software can generally 
import and handle large numbers of individual client 
records. At a minimum & can produce lists and 
summary tables of selected variables. The ability to 
go further depends on the software product. in 
general, there is a trade-off between ease of use and 
flexibility of output. 


For data imported trom an existing management 
information system, a user-friendly, flat file data base 
manager is probably sufficient for most purposes. 
They have the ability to calculate new fields and 
generate crosstabulated tables as well as routine 
summary statistics. These programs assume that 
users are responsible for producing their own output, 
and they make & easy. 


Relational dais base managers typically offer more 
flexibility in refining your data and designing output, 
bul they generally require a higher level of 
programming skill. Ohm, thoy are oriented toward 
development of fixed systerns with a heavy emphasis 


On pre-programmed reports. if your primary need is 
to enable staff to generate ad hoc reports on an as- 
needed basis, a relational data base manager may 
not be the best choice. 


Are you thinking about purchasing a data base 
program to supplement your MIS? The match 
between the computer skilis of your staff and the skill 
level required by the product you are considering is 
important. 


STATISTICAL SOFTWARE 

For staff with formal training or work experience in 
social science research, statistical software is likely to 
be an old friend. For other staff, t may be a complete 
mystery. Because of the power it gives users, ft is a 
tool that can help fine tune data and produce a 


trained in data analysis, as well as at ease with 
comnuters. 


Simple and sophisticated analyses are easy to 
perform with statistical software. Many more options 
are available to the user in structuring their data and 


output. 
DATA MANIPULATION 
Having access to software which allows users to 


all the answors. As users begin to work with 
sophisticated new tools, two facts are likely to 
become apparent. 


First, the data in fies designed to support the 
generation of canned reposts and look-up screens 
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cannot be used “as is". Second, higher level 
computer skilis must be learned before the potential 
offered by the new software can be realized. 


The MIS staff which thinks that — wil inssen the 
demands placed on i by providing users with data 
manipulation capability can easily find that f has 
simply created new demands. That doesn't mean 
that States ami SDAs should not attenpt to provide 
users with flexible analytic capability. it does mean 
that they should be prepared to provide more 
support, not less. 


Missing Vaives. One of the issues which has a 
substantial effect on the ability of users to manipulate 
follow-up data is how missing values are handled. 
“Missing data” refers to situations where the 
information which should be entered into a field is 
unknown. Several types of missing data occur in 
follow-up, and it is important for users to be able to 
tell them apart. They include: 


a Missing Interviews. The entire follow-up 
interview is missing for terminees who are 
not included in the follow-up survey or who, 
although included, were rot contacted. 


Ss Lack of Response. People who are 
‘terviewed may refuse to respond to 
indiidual questions or say that they do not 
know the answer. There can be no missing 
values for mandatory follow-up questions. 


a inapplicable item. An individual's response 
to one Rem may make another tem 
inapplicable. For example, people who were 
not empsoyed at follow-up or did not work 
during the 13-week prriod would not be 
asked questions regarding labor force 
participation because & is already clear that 
they were not in the labor force. 


Most users are unfamiliar with missing data, and 
software packages differ in the ease with which they 
handie t. MIS staff can assist users by including a 
selection field for completed interviews, using a 
consistent coding system for missing data, and 
replacing the missing data code with the logical 
response in filtered fields. 


MIS staff can expect that users will need training, 
sample programs, and continued support as they 
work with follow-up data. 9 3 


Social Programs at the State and Local Level: The JTPA Evaluation 


Michigan, 1990. This 


Quide looks at a 


. Upjohn ; Kalamazoo, comprehensive 
of evaluation strategies Full of useful ideas and materials. A must for JTPA evaluators. 


Herman, Joan L., Ed. The Program Evaluation Kit, Sage Publications. This is a practical step-by-step set of 
nine volumes for practical guidance to evaluating social programs. 


Rossi, Peter H. and Freeman, Howard E. Evaluation: A Systematic Approach, 4th Edition, 1989. 
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Validating postp:ogram data is an important issue. Learn if this is something you should 
or should not be doing in your area. 


Reference week earnings are a key postprogram indicator. Learn why you can not use 
information outsice the “window.” 


if you contract for follow-up, learn how to improve your RFP. What areas need to be 
highlighted and emphasized. 


Special populations in follow-up may require special attention. We present a policy for 
dealing with special populations. 


re 
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In this module we review the importance of validating your postprogram data. Also, we will look at reference week 
earnings and explain the importance of collecting data for the reference week exclusively. The special attention 
required when doing follow-up with special populations and a suggested policy for dealing with them is presented. 
Ways of improving your RFP for follow-up contractors and areas of special concern are highlighted. 


POSTPROGRAM DATA: A SAMPLER 


| EVALUATION QUESTIONS FOR | 


Module 6 discusses some questions you might want 
tc ask about your program. Here are other ideas: 


Analysis of postprogram data in support of program 
planning might begin by looking at outcomes by 
target group and activity. Perhaps predictable 
ditferences are found: Dropouts work less than high 
school graduates, women earn less than men, and 
employment is higher among participants in OJT. 
But findings like these don't tell us anything we don't 
already know, and they didn't help us design better 
programs. After all, we're not going to stop serving 
dropouts and women; and we know that the 
outcomes of OJT are due, in part, to how participants 
are enrolled in OUT in the first place. 


The questions that are useful are the ones which lead 
us to ideas on how to improve outcomes and lessen 
differences, given that our client population will 
remain largely the sarne. Here is a saniple of 
questions which go beneath the surface and which 
postprogram data cun help you answer. Some of 


a Among dropouts, what activities and services 
are associated with higher employment rates 
at follow-up? Did GED completion 
contribute to higher levels of employment 
and earnings for dropouts? What about the 
new, basic skills brush-up program? Does it 


have an impact on employment rates? 


a What were the “best” placement jobs for 
women, based on retention, earnings, and 
benefits? How aid the workforce retention 
and earnings of women who entered non- 
traditional jobs compare to the workforce 
retention and earnings of women who 
entered other jobs? 


a What factors are associated with 
employment and earnings at follow-up 
among welfare recipients? For example, 
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were mandatory participants less successful 
than voluntary recipients? Were long-term 
recipients less successful than short-term 
recipients? How was number and age of 
children related to employment and earnings 
at follow-up? To what extent are welfare 
recipients working in jobs which provide 
health care benefits? What do our post- 
program data tell us that we need to share 
with our welfare agency so as to provide 
more effective service to these clients? 


a Were there meaningful differences between 
service providers? To what extent did 
differences between individual service 
providers account for apparent differences 
between results in the employment, earnings, 
and workforce retention of participants? 
How do these differences relate to what we 
know about the way in which services were 
provided? What do the postprogram 
outcomes of participants who received 
innovative services tell us about those 
services? ok 


| VALIDATION | 


The 1990 TAG. spends 13 pages discussing 
validation issues. Validation is a complex issue that 
takes many forms. Experience has now 
demonstrated the problems with validation and 
T.A.G.* will attempt to shed some light on these. 


T.A.G.: pp. 87-99. 
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ACCURACY OF DATA 


USDOL regulations require that follow-up data be 
provided directly by the terminee. No surrogate 
sources, such as a spouse, are permitted. 
Consequently, every reasonabie effort should be 
taken to make certain that the questions and the 
dates referenced are well understood by the 
terminee during an interview. 


Terminees have been known to provide biweekiy 
instead of weekly income, net instead of gross pay, 
gross pay without overtime or bonuses, and income 
received instead of earned. The mandatory 


questions are not as simple as they appear. 


Yet, in order to have and maintain consistency 
across all SDAs, only terminee-reported data are 
acceptable, regardiess of any other information the 
State or SDA may have or receive from a third party. 


INACCURACY OF DATA 

When a State or SDA has information contrary to 
what the terminee reported, only the terminee can 
provide a resolution to the discrepancy. 


T.AG.: pp. 97-98. 
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Example -- Suppose you routinely attempt to verify 
reported earnings with employers. A discrepancy is 
found. Let’s say the employer reported wages 20% 
higher than the terminee. 


This suggests the terminee reported net instead of 
gross earnings. Within the five week window allowed 
for the collection of this information, you may 
recontact the terminee to clarify the situation. 


Of course, you might also find out that the employer 
reported earnings about half as high as did the 
termines. This suggests that the terminee earned 
tips, had a second job or reported biweekly earnings. 


Equai attention should be given to any errors that are 
discovered concerning discrepancies in wage 
information. Whether under or over-reported, such 
discrepancies can inject bias into your data. 


A concern over impropriety is generated when SDAs 
collect their own postprogram data on which 
performance measures are to be based and by which 
incentive funds will be distributed. 


tt is appropriate for States which permit SDAs to 
conduct the’ own follow-up surveys to institute an 
independent validation of data to protect the integrity 
of the performance standards system and to ensure 
that proper accountability is being maintained. It 
should also be assured that consistent practices are 
being followed by all SDAs. 


TIMING OF VALIDATION 


The five week interviewing window was imposed 
because of the well-documented “memory effect” in 
survey behavior. While there are several memory 
effects, of greatest concern is that people forget 
exponentially over time. If validation is not timely, 
then differences measured may be memory effects 
only. The data collected closest to the reference 
week would carry greater credibility. 


Independently collected data assumes that the data 
collection contractor has no stake or interest in the 


monitor the data collection process to ensure that 


professional standards of operation and data 
coliection are being maintained. 


IMPLICATIONS OF VALIDATION 


Follow-up data, as have been discussed throughout 
T.A.G.*, can have a major impact on the 
management of the JTPA program at the State and 
SDA level. it is important that the information 
available to planners, decision makers, and any other 
users be as accurate as possible. Validation is just 
one part of the process that ensures this will happen. 


RESOLVING CONFLICTS 


When you get conflicting data, you have to determine 
the reason. Changing data after the fact can only 
occur based on the terminee’s original or revised 
responses. It is most important that the causes for 
discrepancies be treated. 


The Pre-Survey Letter may need to be more specific. 
The questionnaire might need improvements. 
interviewer instructions and training might need to be 
expanded. Whatever you learn from yesterday's 
work should be used to improve tomorrow's. 9 3 


a ‘J REFERENCE WEEK EARNINGS 
mee @ 


a= 


in a continuing effort to improve the quality of your 
data collection effort, we report to you this recently 
overheard conversation. 


"| really don’t think it’s fair. We found someone who 
was working all 13 weeks but he was a trucker who 
didn't carry a load in the reference week so he didn't 
earn anything. We thought we should be able to 
Claim his average week’s earnings * 


No way. This whole thing is set up to 
measure a standard - and that standard is 
what they made in week 13. Your trucker 
was employed, but had zero income. 


“But there’s nothing special about that thirteenth 
week. Not getting any income was just chance. | 
think we should count the average inco™. * 


OK, but what about the por you 
interviewed who only had incc ye in that 
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"Well, another problem is how to deal with school 
teachers. They get paid either over 9 months or 12 
months.° 


week. Are you going to average twelve 
weeks of zero’s with the 13th week? 


“Of course not. That's different. They really had 
earnings that week.” That's a bit different. Remember the 


Sorry, not different. | understand why you 
wamnt the best of both worids, but you have to 
follow consistency rules. if you don't, it's 
just a numbers game, and at the end of it 
you don't really know what your program is 
doing. You might end up with great looking 


question asks about doing work for pay that 
week. if they had received income for work 
done months ago, you would have zero 
earnings even if they got checks. The real 
question is whether they are employed, and 
for that | would want to know when their 
contract expired. if they were under 
contract, | would consider them employed. 


“OK, so what do | do about people who are self- 
“But that's still not fair. We really did help people. It's employed? We have a few of those.” 
just that the kind of jobs we have in our area don't 


always have weekly earnings.” Once again, you find out if they were 


But don't you think that is an important thing 
to let people know? That's something that 
should be put into the planning, and should 
be part of the calculation for your 
performance standards. If you set yourself 
up to artificially inflate the earnings measure, 
you really end up promoting a false and 
distorted view of reality and only hurting 
yourself. 


“| guess | hadn't thought of it that way.” 


Don't feel bad. A lot of people haven't. The 


real issue is not whether you have "good" 


data or ‘bad" data, but whether you have 
accurate data. !'ve seen your data and the 
economic forecasts. You have some major 
challenges in front of you. Artificially good 
data is not going to help you make your 
case. 


“But this still doesn’t make my program look very 
good.” 


But it is an eccurate reflection. Remember, 
you still record the individual as working in 
the reference week and that is positive. You 
just have zero earnings. You'd have the 
same situation with people working on 
commission who had a bad week. 


working or trying to get contracts or 
whatever they do. Even if they did not earn 
anything that week, if they were actively 
seeking work for their own company, they 
would be considered employed. Consider 


the following two examples. 


One is a house painter. He is an 


independent contractor and has a three 
week contract with a builder to do a house. 
He was paid 30% up front and the balance 
when he is done. You interview him in the 
second week. He didn't receive any money 
that week but he did earn 1/3 of the total 
contract. Earnings would be net of his 
expenses. That requires your interviewers to 
really be heads up on probing for this 
information and understanding how to 


handle self employed. 


The second example is the same person but 
you interview him in the third week after he 
got paid. You would still only record 1/3 of 
the total contract (or half of the balance 
received) as earnings for that week. 


"That's really complicated, though. These couple of 
mandatory questions aren't as simple as they look." 


The questions are simple enough. It's the many 
different situations that your participants face in the 
workforce that’s complicated. Spend some time 


xy 
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thinking them through. Talk to your interviewers. 
And remember the intent of the questions. The two 
concepts to focus on are working and earning. it's 


Many JTPA programs are designed to assist hard-to- 
serve populations. Some of these populations have 
more difficult barriers to surmount than others. 


One cardinal rule in follow-up data collection is that 
the terminee must provide the information to the 


researcher. No surrogate -- spouse, employer, or 
anyone else -- can be the source of the answers. 


When language barriers are encountered, it is 
permissible to go through an interpreter. The 
cardinal rule romans the same, however, in that the 
terminee must be present, hearing the question, and 
providing the answer which is in turn transmitted to 
the interviewer. 


While certain conditions permit the terminee to be 
eliminated from the sample and universe, guidance 
has not yet been provided on how to deal with 
persons being served by JTPA who have diminished 
capacities. 


TAG.: p. 30. 
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Some JTPA programs deal specifically with large 
numbers of people who are mentally handicapped. 
While some of these people can provide competent 
answers to the follow-up questions, others cannot. In 
particular, some participants may have difficulty 
understanding the reference \veek, knowing hours 
worked, or reporting wages earned. 


if this group of people were to be eliminated from the 
universe, the SDA would be denied follow-up 
information on an important portion of the 
disadvantaged population they were trying to serve. 


f fit 


the accuracy of your data that really improves your 


planning. *% 


To permit the most accurate information to be 


collected on this population, the following procedure 
is recommended: 


A mitigating circumstance related to t*2 rule of 
collecting all follow-up data directly from the 
former participant occurs when the terminee is in 
a@ population which is considered ‘menially 
diminished." Examples include brain damaged 
individuals, mentally retarded individuals, and 
individuals with Alzheimer's disease. Some JTPA 
programs have been designed to specifically 
address the unique employment challenges 
experienced by these people. 


When dealing with ‘mentally handicapped" 
individuals, consider collecting follow-up data 
from a qualified surrogate (i.e., an individual who 
has direct access to relevant supporting 
documentation for their answers). This will most 
often be a spouse, guardian, counselor, 
physician, case manager or therapist. 


When information is collected from a surrogate, 
the following information should be collected and 
maintained with the individual's file: 


Name of surrogate 

Relationship to terminee 

Organization, if applicable 
Telephone contact number 
Date of interview 
Name of the interviewer 


— 


ok 


¥S 
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CONTRACTING FOR FOLLOW-UP 


Over haif of the States that have centralized systems 
use an outside contractor to conduct follow-up. 
Some SDAs which are responsible for follow-up also 
use Outside contractors. 


TAG.: pp. 19-21; Appendix D. 
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The 1990 T.AG. provides you with a brief outline for 
writing an RFP (Request for Proposal) for follow-up 
contractors. The quality of the responses you 
receive may be directly proportional to the quality of 
your RFP. Many States issue RFPs that are much 
more detailed than the example found in the T.A.G. 


Since Appendix D really only deals with what you 
ask, instead of how to evaluate the answers, this 
section will give you some things to consider. litem 
numbers refer to specific items in the 1990 TAG. 


RFP for ease of comparability. 


2. Response Deadline 


The standard response time is 4 weeks. If you have a 
pre-proposal conference, you should allow 4 weeks 
after the conference. You want a competent, well- 
considered proposal. Regardiess of how much you 
like your Current contractor, you should allow other 
vendors a fair chance to bid. Make certain that you 
allow enough time in your review process for you to 
adequately verify the claims of bidders. Ask for and 
check out references. 


3. _ Eligible Bidders 


USDOL encourages free and open competition for 
these contracts. While your State may give 
preference points for minority, woman-owned, or 
home-State-based companies, we urge careful 
consideration of any limitations that may provide 
your citizens with low er quality services. 


4. Narrative instructions 


Your task of assessing competing proposals will be 
difficult enough. You ask for some standardization of 


responses to facilitate this process, while still leaving 
room for individual initiative and ideas. it is 
appropriate to disqualify or penalize bidders who 
ignore what you ask for. if tt happens at the proposal 
stage, what will happen in the project? 


6a. interviewers and Supervising Staff 


With respect to getting names and qualifications you 
should be most interested in supervisory staff. 
interviewers will probably change over time. Of 
greaiest interest should be the skilis and experience 
of the interviewing staff. These are really the key 
people in the process. The staff must be capable of 


The interviewer is the person in direct cortact with 
the terminees. This person represents you, ths client, 
as well as the research company. The quality of their 
want to ask about their training -- length and content, 
how they are evaluated internally, and how they are 
paid. Payment by completed interview can be an 
incentive to churn and get the easy cases. 


6b. Schedule For Conducting Interviews 
Here you are looking for hours and days of op- 
eration. How will the bidder staff this? Ask them. 


6c. Conduct Interviews 


This is a key section. Does the bidder have a plan? 
Do they understand the problems with co=ducting 
scientifically valid surveys with this population? 
Provision to accept calls from terminees is critical in 


achieving required response rates. How will this be 
handled? What contact will the contractor have by 


mail with respondents, and with what timing? 


Record and Verify Data 


"el 


Recording contact attempts is discussed in Module 3 
in which we discuss the value of merging into your 


Xl 
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data base key operational statistics. Double entry 
procedures are standard in scientific data collection 
provided the entry is conducted by two separate 
people. Double entry really has nothing to do with 
validity but does help ensure entry accuracy. 


Accurate information involves numerous quality 
control steps. The first step is the extent and quality 
of interviewer training. The second is how well 
interviewers are trained to the survey and what 
supporting materials they have. The third is the 
interviewer editing procedures for their own surveys. 
Fourth is the coding and quality control process. 
Supervisory quality control procedures are fifth. 
Then comes data entry and entry verification. A 
sophisticated entry software program permits on- 
entry validation (range checking and error checking). 


Co:..puter Assisted Telephone interviewing systems 
(CATI) should be sophisticated enough to permit 
within-interview correction procedures and automatic 
back-up in case of system failure. 


There are many ways to do these steps well. The 
bidder should explain its process in detail...and 
convince you they know what they are doing. 


6e. Minimum Response Rates 


__ bs 
ee 


Achieving a 50% response rate for all groups can be 
accomplished with relative ease. Getting to 70%, 
especially with groups not employed at termination, 
requires perseverance. What procedures wili be 
employed to achieve the cost-effective maximum 
response rate? You want to see details of their 
location strategy. The bidders’ answers will give you 
insight as to their sensitivity to the special problems 
encountered in follow-up data collecticn. 


Quality interviewing is important, but the most 
expensive part of the process is location. A proven 
track record in follow-up interviewing or interviewing 
of other disadvantaged populations is a definite pius. 


You can lll afford a bidder who will take 8 to 10 weeks 
to get up to speed on this critical issue 


6t. Validation of Responses 
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Validation is an ambiguous issue. if intake of 
information by the SDA goes through validation of 
recording and entry, more confidence can be taken 
with using birthdate as a validation question. Review 
the 1990 T.A.G. and the front of this module. 


6g. Reporting 

The information in Appendix D focuses upon the 
mandatory questions and performance measures. 
An understanding of the non-response bias 
adjustments should be demonstrated by the bidder if 
the bidder will be required to report this information 
using these procedures. 


With respect to reporting, these items would be bare 
minimums. if you desire analytic reports, this should 
be discussed with your contractors. Considerable 
variation can be entertained. A good rule, if analytic 


reports are requested, is to ask for previous reports 
and to have this option itemized. 


7a. Sampling 
MOD 2, pp. 9-11. 
| a 


Make sure the bidder understands how to select the 
welfare supplemental sample, if the bidder is 
responsible for sampling. Timing and delivery of the 
sample, as well as information contained therein, will 
vary widely among States. Be certain to clearly 
specify responsibilities and to ascertain whether the 
bidder has the appropriate technical capability and 
facility 


7b. The Questionnaire 


Decide whether you or the contractor will provide the 
questionnaire. if the contractor is to provide the 
questionnaire you will want to be assured that they 


K} 
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have the background, experience and capability to 
design this in a professional manner. Always ask to 
see examples of other questionnaires. 


Other Bidding Concerns 


Writing the RFP is not half as difficult as evaluating 
the content of your bids. Module 6 lists the different 
data collection options followed by all of the States. 
Network with your colleagues for ideas on how to 
improve your RFPs and evaluation systems. 


Encourage your bidders to go beyond the minimum 
requirements and to propose enhancements which 


would improve your entire system. 


Several ditierent payment options would be available. 
The common ones are fixed price, cost 
reimbursement, price per sampled terminee, price 
per completed interview as well as hybrid 
arrangements. if you use a price per completed 
interview or intend to impose penalties for response 
rates under 70%, any experienced contractor will 


expect you to provide details on your operations. 


For example, the experienced bidder knows that the 
quality of location information provided to them will 


Gwectly affect the ease or difficulty of location which 
will affect their quote. if you are providing two phone 
numbers for only 40% of your terminees, and no 
phone numbers for 20% of them, the bidder knows 
that location will be extremely difficult. Their price will 
refiect that knowledge. 


Evaluating your different bids should be done as 
objectively as possible. But as most readers 
understand, the chemistry between bidder and 
contractor is also important. This chemistry is 
difficult to quantify in an objective rating scale. 
Although this is subjective, allow some rating points 
for how good a match you feel you have. 


if you are seriously considering a bidder you have no 
experience with, a face to face meeting (preferably at 
their work site) will provide you with the opportunity 
to meet the key staff and assess whether they can 
deliver what they promise. 


It is likely that your bids will differ. The cost of the bid 
must be balanced against the quality and quaniity of 
service proviried to you. Buying the lowest bidder 
without adequately factc “ng in what you get is not 
recommended. You looking for the best 
cost/value relationship. © 4 


The State of Minnesota has been working on a validation procedure. Contact Larry Eisenstadt, Dept. of Jobs 


| and Trairiing, State of Minnesota, 690 American Center Building, St. Paul, MN, 55101; (612) 396-6073. 


ch West is the statistical contractor for USDOL. For the really detailed questions, you can contact him at: 
Mr. Rich West, Vice Presicent; Social Policy Research Associates; 200 Middlefield Road, Suite 100, 


Menlo Park, CA 94025; (415) 617-8627 


a We present a “study guide” that can be used to evaluate and improve your follow-up 
system. While we do not have answers to all cf the questions posed in the guide, we will 
direct you to areas that are appropriately covered in the 1990 T.A.G. and T.AG.*. 


XY 


T.A.G. + 


Supplement to the Technical Assistance Guide for JTPA Follow-up and Validation — Module 12 - 1992 


Published by the Employment and Training Administration, U.S Department of Labor, Fn NS629, 200 Constitutics: Avenue, N.W., 
riashington, 0.C. 20210 - (202) 219-5782 


in this module we attempt to summarize the information presented in the previous T.A.G.* modules. What we 
have produced for this module is a combination of the questions that were raised during our site visits and the 
knowledge that was obtained in the process. Key areas in the follcv-up process were identified and provided the 
basis for our approach. 


The summary is presented in outline form using questions that are appropriate to the follow-up and evaluation 
system at all levels. Where appropriate we have referenced both the 1990 T.AG. and T.A.G.*. References to the 
respective T.A.G.s are located beneath the questions with the 1990 T.A.G. located to the left and T.A.G.* located 


it ts our intention that this module be used as a “study guide” for reviewing the components and p:.cesses that 
exist in your follow-up system. The study guide can be used in part or whole. You can use the stucly guide as a 
year-end assessment to determine where you have been, what areas, if any, require modification or change, and 
where you want to take your program in the coming year. You can also scan to a specific topic in order to review 


We have also produced a merged index that can be used to locate information in both the 1990 T.A.G. and 
T.A.G.*. References to T.A.G.* will appear in bold in order to differentiate them from ihe 1990 T.A.G. 


Prepared for the U. S. Department of Labor, ETA by Appropriate Solutions, inc., Columbus, Ohio. Contract #99- 1-3654-72-078-01 
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A STUDY GUIDE 
FOR FOLLOW-UP AND EVALUATION 


A. Do you wish to interview all terminees or a sample? 
1. What are the minimum sampling requirements? 
1990 TAG. pp. 23-25 LAG.+ MOD 2, pp. 9-10 
2. Why consider oversampling or interviewing all terminees? 
1990 TAG. pp. 25-27 LAG.+ MOD 2, pp. 10-11 
3. Which additional populations can be included in your follow-up? 
1990 TAG. pp. 10, 11 TAG.+ MOD2,p. 10 
B. Who provides the terminee data to your organization? 
C. With what frequency are the data provided? 
1990 TAG. p. 27 
D. The data are provided in what format? 
E. if applicable, how is your MIS structured? 
1990 TAG. pp. 67-69 
F. What data are availabie on your MIS? 
G. What data are provided to follow-up operations from your MIS? 
1990 TAG. pp. 67-68 LAG.+ MOD 3, pp. 17-18 
H. Are late reported terminations a problem for you? 
1990 LAG. p. 27 LAG.+ MOD 7, pp. 53-55 


ll. DESIGN AND STRUCTURE OF THE SURVEY QUESTIONNAIRE 


LAG.+ MOD 1, pp. 4-7 
MOD 2, pp. 11-12 


A. Are you collecting information beyond the mandatory questions? 
1990 TAG. pp. 8-10 LAG.+ MOD 1, pp. 45 
App. C 
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1. Are you gathering attitudinal information from terminees about their 
program experiences? 


1990 TAG. p. 8 ITAG.+ MOD 1,p.4 
2. How would your terminees assess the services received? 
3. Are benefits provided by employers? 
4. What is the rate of the welfare dependerice among terminees? 
1990 TAG. p. 9 
5. What other issues should be included in your questionnz.e? 
How important is the wording in your questionnaire? 
1. Are the questions posed from the terminees’ perspective? 
LAG.+ MOD 1, p.6 
2. Do the questions enabie your respondents to answer accurately? 
LAG.+ MOD 1, p.5 
3. Will the wording lead to a socially acceptable response? 
4. Do your questions remain neutral and unbiased? 
LAG.+ MOD 1,p.5 
How does the order and placement of questions affect your survey? 
TAG.+ MOD ~,p.5 
1. Do any attitudinal questions precede the mandatories? 
1990 TAG. pp. 8,49 
2. Does the order of your questions bias any of the responses? 
LAG.+ MOD 1, p.5 
3. Is the logic flow of your questionnaire biesed? 
LAG.+ MOO 1, p.5 
is your cjuestionnaire formatted correctly? 


LAG.+ MOD 1, pp. 67 
MOD 2, pp. 11-12 


1. ts your questionnaire easy to read and administer? 


LAG.* MOD 1, pp. &7 
2. Does your formatting promote the accureie recording of responses? 
TE G+ MOD 2, pp. 11-12 


BLANK PAGE 
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3. Does the questionnaire have enough blank space to avoid seeming 
crowded? 


TA.G.+ MOD2, pp. 11-12 
4. Have you included enough space for additional notes to be recorded? 
TA.G.+ MOD2, pp. 11-12 


il. INTERVIEWING AND THE DATA COLLECTION PROCESS 
A. How does your organization conduct follow-up? 


1 T.A.G. pp. 17-19 T.A.G.+ MOD6, pp. 46-47 
1. Should data collection be centralized or decentralized? 
1990 T.A.G. pp. 17-16 
2. Should data collection be conducted in-house or contracted out? 
1990 T.A.G. pp. 17-20 T.A.G.+ MOD 11, pp. 86-88 
App. D 
a. Is there a validation mechanism in place? 
1990 T.A.G. pp. 20-21 T.A.G.+ MOD 11, pp. 82-83 
pp. 68-69 
pp. 87-89 
pp. 97-99 


b. If decentralized, how good is State coordination in data collection? 
3. What should you include in a Request for Proposal? 
1990 TAG. App.D 
4. How much time have you allowed for a response? 
TA.G.+ MOD 11, p. 86 
5. What things are you looking for in a bidder’s response? 
TA.G.+ MOD 11, pp. 86-88 
a. Does the bidder have relevant experience? 
b. Will the bidder be capable and responsible for sampling? 


1990 TA.G. pp. 28-31 TA.G.+ MOD 2, pp. 9-11 
App. E MOD 6, pp. 45-46 


MOD 8, pp. 59-60 


4. 
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c. Does the bidder have experience in questionnaire design? 
TAG.+ MOD 1, pp. 47 


MOD 2, pp. 11-12 
d. Are the interviewing capabilities of the bidder adequate for your 
needs? 
TAG.+ MOD 3, pp. 19-21 
| MOD 5, pp. 33-35 
e. What MIS capabilities are possessed by the bidder? 
1990 TAG. pp. 67-69 TAG.+ MOD 10, pp. 78-79 
f. What types of reports, if any, will the bidder be expected to produce? 
1990TAG. pp. 77-85 TAG.+ MOD 10, pp. 73-77 
g. How does the bidder intend to meet and/or exceed the minimum 
response rates? 
1990TAG. p.7 TAG.+ MOD 3, pp. 18-23 
MOD 4, pp. 26-31 
MOD 11, p. 88 


h. What type of payment option is the most suitable for your situation? 


TAG.+ MOD 11, p. 8 
8. Why are interviewers important to your follow-up process? 


TAG.+ MOD5, p. 33 
1. What types of skills should your interviewers possess? 


LAG.+ MOD 4, pp. 31-32 


MOD 5, pp. 34-35 
MOD 5, pp. 37-40 
2. What components should be included in vour interviewer training? 
1990 TAG. pp. 59, 61, 
pp. 62-65 LAG.+ MOD 5, pp. 36-39 


3. How will you supervise and evaluate your interviewers? 

1990 TA.G. pp. 59, 66 
4. What elements are important to establishing your interviewing facility? 
5. Is the interviewing staff adequate for your needs? 


LAG.+ MOD 3, pp. 19-21 


45 


T.A.G.* — Module 12 — Page 94 


6. How do you handle interviewing special populations? 
LAG.+ MOD 11, p. 84 
C. What operational factors are necessary for follow-up interviewing? 
TAG.+ MOD 3, pp. 19-21 
1. What is the organization of your follow-up process? 
2. What time of day are cails taking place? 


1990 TAG. p. 44 LAG.+ MOD 3, p. 20 
3. Which days of the week are calls taking place? 
1990 TAG. p. 44 TAG.+ MOD3, p. 20 


4. Is an 800 number provided for call-ins? 


1990 TAG. pp. 41, 44 LAG.+ MOD 3, pp. 18-19 
5. Do you accept collect calls? 


1990 TAG. p.41 TAG.+ MOD 3, pp. 18-19 
6. What type of training would be beneficial for your staff? 
1990TAG. pp. 59-66 TAG.+ MODS 


7. How many weeks are in your interviewing window? 
1990 TAG. pp. 33-35 TAG.+ MOD 3, p. 20 
8. How do you handle potential refusals to the interview? 
1990 TAG. pp. 52, 56 TAG.+ MODS, p. 38 
9. What issues are associated with paying terminees? 


LAG.+ MOD 3, pp. 23-24 


10. y Sry nen is there to feed problem information back to the State or 


D. How is your terminee call record designed? 
1990 TAG. p. 47 LAG.+ MOD 3, pp. 17-18 
1. What terminee information are you providing in the call record? 


1990 TAG. pp. 37-38 LAG.+ MOD3, p. 18 
pp. 46-47 


2. What kinds of information are recorded on the call record? 


LTAG.+ MOD 3, pp. 17-18 
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4. Have you allowed space to record notes and update information? 
TAG.+ MOD 3, pp. 17-18 
5. Are you analyzing the information from your call records? 
LAG.+ MOD 3, pp. 21-22 
E. What pre-contact procedures are used? 
1990 TAG. pp. 41-43 LAG.+ MOD 4, pp. 25-26 
1. Are you using a pre-survey letter? 
1990 TAG. pp. 41-42 LAG.+ MOD 4, pp. 26-27 
a. When is the pre-survey letter sent? 
1990 TAG. p.41 TAG.+ MOD 4, p. 29 
b. Who does the pre-survey letter come from? 
LAG.+ MOD 4, pp. 26-27 
c. Are address correction and forwarding requested on your envelopes? 
1990 TAG. p. 41 LAG.+ MOD 4, p. 29 
d. Are you Including a business reply postcard for updated contact 


1990TAG. p. 41 TAG.+ MOD 4, p. 29 
e. a envelopes distinctive and easily distinguishable from junk 
TLAG.+ MOD4, p. 29 
F. How are you utilizing telephone pursuit in your follow-up process? 
1990 TAG. pp. 44-45 TAG.+ MOD 4, pp. 31-32 
MOD 5, pp. 37-38 
1. How many attempts are made at contacting a terminee by telephone? 
1990 TAG. p. 45 LAG.+ MOD 3, p. 22 
2. Are your contact attempts varied by time and day? 
1990 TAG. p. 44 TAG.+ MOD 3, p. 20 
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T.A.G.* — Module 12 — Page 96 


3. Are you using telephone books in location and pursuit strategies? 


1990 TAG. p. 45 TAG.+ MOD 4, pp. 28-29 
4. Are calls being made to information operators? 
1990 TAG. p. 45 TAG.+ MOD 4, p. 29 
5. Have you considered using criss-cross directories? 
1990 TAG. p. 45 TAG.+ MOD 3, p. 23 
MOD 4, p. 30 
6. Do you attempt to reach the terminee through employers? 
1990 TAG. p. 45 TAG.+ MOD 4, p. 31 


7. Are attempting to contact other sources such as case workers, SDAs, 
halfway houses and the criminal justice system? 


1990 TAG. p. 38 
8. Are you leaving messages whenever possibile? 
G. How are you using mail pursuit in your follow-up process? 


1990 TAG. p. 52 LAG.+ MOD 4, pp. 29-30 
pp. 57-58 


1. How many attempts are made at contacting terminees by mail? 
TA.G.+ MOD 4, pp. 29-30 
2. Does your mail encourage terminees to call in? 
1990 T.AG. pp. 52, 57 LAG.+ MOD 4, pp. 27, 29-30 
3. Does your organization utilize a mail survey? 


1990 TAG. pp. 49, 52 TAG.+ MOD 2, pp. 13-15 
pp. 57-58 


a. is the mail survey easy for terminees to accurately complete? 
TLA.G.+ MOD 2, pp. 13-15 

b. Is a postage-paid envelope included with your mail survey? 
TAG.+ MOD 2, p. 14 

c. When is a mail survey used or sent out? 


LA.G.+ MOD 2, pp. 13-15 
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IV.Post-SURVEY OPERATIONS WITHIN YOUR SYSTEM 


A. How does your organization handie data entry? 
1. What quality control procedures has your organization implemented? 
1990 TAG. pp. 68, 69 LAG.+ MOD 7, pp. 51-52 
2. Does data entry employ two passes with verification? 
3. When are you entering your data into the computer? 
4. Are any operational and location statistics being entered? 
LAG.+ MOD 3, pp. 21-23 


B. Does your organization perform data analysis? 
1. ts follow-up data merged with MIS? 


a. What data elements are you merging with the follow-up data? 
TAG.+ MOD6, pp. 41-42 

| 2. What type of computer hardware is used? 

3. What type of computer software is used? 

1990 T.A.G. App.H LAG.+ MOD 10, pp. 78-79 

4. What training is provided for those involved with data entry and analysis? 

5. Have responsibilities been clearly defined and assigned? 

6. What information from follow-up is available to SDAs? 


1990 TAG. pp. 13-15 
pp. 77-86 


a. is data provided in aggregate or individual form? 


V. UTILIZATION OF THE FOLLOW-UP DATA 
A. Do you know how to complete the mandatory reports for USDOL? 


1990 TAG. pp. 71-76 LAG.+ MOD 7, pp. 53,55 
pp. 77-79 MOD 8, pp. 60-63 
App. 8 


1. Who has the responsibility for generating the mandatory reports? 
2. When are the mandatory reports completed? 
3. How capable are your SDAs in completing these reports? 


47 
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B. Are you producing special reports using follow-up data? 
1990 TAG. pp. 80-84 TAG.+ MOD6, p.43 


1. Are you producing tabular reports? 


1990 TAG. pp. 80-82 
2. ae yen proms Cee , reports which look at the information 
to patterns or r ips? 


3. Are , which attempt to i 
rlatonsnipe and derveprogant ang potcy impact and 
recommendations? 


4. Are you producing methodological reports, which explain proced.es and 
processes you might use in the follow-up and evaluation process? 


5. Are operational data being analyzed for any of your methodological or other 
reports? 


6. Who is responsible for producing special reports within your organization? 
7. How many different special reports should be produced? 
8. What audiences are you producing your special reports for? 
1990 TAG. pp. 13-16 LA.G.+ MOD7, pp. 49-50 
9. How will those special reports be distributed? 
C. Are you using follow-up data for planning purposes? 


LAG.+ MOD9, pp. 67-71 
MOD 10, pp. 73-77 


1. How is follow-up information being reported to SDAs? 


2. How is information from follow-up being integrated into planning or other 
management processes? 


3. Are you facilitating the dissemination and use of follow-up information 
through training seminars and workshops? 
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ACCURACY, DEFINED, pg 88 
ACCURACY AND VALIDITY OF DATA COLLECTION, 
IMPORTANCE OF, pg 7 
ACCURATE AND COMPLETE INFORMATION, 
INTERVIEWING TO OBTAIN, pp 51-52 
ACCURATE, VALID FOLLOW-UP, GENERAL 
GUIDELINES TO PROMOTE, pp 20-21 
ACCURATE, VALID FOLLOW-UP DATA, 
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Process of, pp 74-77 
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CALLS, INCOMING 
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23, 27-31 
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Crosstabulation, pg 76 
Frequency Distributions, pg 76 
identifying and Resolving Discrepancies in, pp 97- 

98 


inaccuracy of, pg 82 
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SDA-Collected, pg 83 
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pg 50 
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Timing, pp 34-36 
DEFINING THE JTPA SYSTEM, pp 2-3 
DEFINITION OF 
“Accuracy”, pg 88 
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Terms Necessary for Completion of Reports, 
Appendix A- pp 25-29 
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EMPLOYMENT RATE AT FOLLOW-UP, pg 5, 
pp 66-68 
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EMPLOYMENT RATE AT FOLLOW-UP (cont'd) 
Formula for Predicting, pg 70 
Fulltime, pg 68 
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Exhibit 4.1, pg 95 
ERROR 
in Survey Data, pg 57 
In individual Responses, pg 58 
in Group Data, pg 59 
Suggestions and Steps for Minimizing, pp 63-64 
EVALUATING, INTERVIEWERS, TRAINING AND 
SUPERVISING, pg 59 
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pp 68-69 
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Appendix B- pg 14 
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FOLLOW-UP INTERVIEWS, CONDUCTING, 
REQUIREMENTS FOR, pg 49 
FOLLOW-UP PROCESS, INVOLVING JTPA STAFF, 
pg 16 
FOLLOW-UP QUESTIONNAIRE, EXAMPLE OF, 
Appendix C, (8 Pages) 
FOLLOW-UP RESULTS, IMPORTANCE OF 
COMMUNICATING, pg 15 
FOLLOW-UP STATISTICS, CALCULATING, pp 71-76 
FOLLOW-UP SYSTEMS, TYPES, Exhibit 1.2, pg 17 
FREQUENCY DISTRIBUTION TABLES, pg 76 
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GUIDEUNES, GENERAL, TO PROMOTE ACCURATE 
AND VALID FOLLOW-UP, pp 20-21 
HOURLY WAGE 


IMPORTANCE OF 
Accuracy and Validity of Data Collection, pg 1 
Accurate, Valid, Follow-up Data, pg 89 
Communicating Follow-up Results, pg 15 
INCOMING CALLS (See CALLS) 
IN-HOUSE FOLLOW-UP, ADVANTAGES OF, pg 19 
IN-HOUSE VERSUS CONTRACTED FOLLOW-UP, 
pg 19 
INFORMATION, INTERVIEWING TO OBTAIN 
COMPLETE AND ACCURATE, pp 51-52 
INFORMATION ON TERMINEES, REQUIRING 
PRIMARY AND BACKUP, pg 38 
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INFORMATION OPERATORS, pg 29 
INSTRUCTIONS 
For Completing the JTPA Annual Status Report 
(JASR), Appendix A-pp 2-14 
For Completing the Worker Adjustment Program 
Annual Program Report (WAPR), 
Appendix B-pp 2-13 
INTERVIEW 
Acceptable Probes, Exhibit 2.10, pg 55 


INTERVIEWER EVALUATION, SAMPLE, Exhibit 2.15, 
pg 66 

INTERVIEWER ORIENTATION GUIDE, SAMPLE, 
Exhibit 2.14, pp 62-65 


Training Outline for, Exhiblt 2.13, pg 67 
INTERVIEWERS, IMPORTANCE OF, pg 33 

Successful, Skills of, pp 34-35 
INTERVIEWING 

Acceptable introductions, pp 31-32 

Acceptable Probes, Exhibit 2.10, pg 55 

Basic ingredients, pp 19-21 

Benefits of Centralized, pg 18 

Clarifying Questions, pg 51 & Exhibit 2.9, pg 54 

Converting Potential Refusals, pg 52 & Exhibit 

2.11, pg 56 

Establishing Rapport, pg 51, pp 31-32, 36-37 
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pp 51-52 
INTERVIEWING WINDOW, REQUIRED, pg 33, pg 15 
INTERVIEWS, FOLLOW-UP, REQUIREMENTS FOR 

CONDUCTING, pg 49 
INTRODUCTION TO JTPA FOLLOW-UP, pg 7 
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General instructions, Appendix A- pp 1-2 
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Purpose of, Appendix A- pg 1 
JASR Section Ili Follow-up information, Extib& 3.1, 
pg 78 
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INSTRUCTIONS (1989 JASR), Appendix A, 
(31 Pages) 
JTPA FOLLOW-UP, INTRODUCTION TO, pg 1, pg 2 
JTPA PARTICIPANTS, PERFORMANCE MEASURES 
OF, pg 7 
JTPA, ATTITUDES AND OPINIONS REGARDING, 
pg 8 
KEY ACTIVITIES, SUMMARY OF, Exhiblk 2.3, pg 36 
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LABOR FORCE NON-PARTICIPATION RATE, pg 68 
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pp 16, 85-86 

LOCATING AND CONTACTING PARTICIPANTS, 


MAIL LETTER AND QUESTIONNAIRE, SAMPLE, 
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MAIL QUESTIONNAIRE, USING A, pg 52, pp 13-15 

MAIL SURVEY, TIMING OF, pg 35, pp 29-30 
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MAXIMIZING RESPONSE RATES, IMPORTANCE OF, 
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MESSAGE NUMBER, DEFINITION OF, pg 26 
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QUESTIONS, Exhibit 2.7 pg 50 
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IMPORTANCE OF, pg 7 
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Appendix A- pp 17-19 
Appendix B- pp 17-18 
Determining, pg 28 
Exhibit 2.1, pg 25 
mMiS 
Potential Problems of “Spring Cieaning’, pg 46 
Setting up for Follow-up, pp 67-69 
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MOTIVATING 


NON-PARTICIPATION, LABOR FORCE, RATE, pg 69 
NON-RESPONSE BIAS, pg 7 
Adjusting Follow-up Statistics for, pg 72 
Procedures to Adjust for, Appendix G (6 pages) 
Worksheet to Adjust Follow-up Performance 
Measures for, Exhibit G.2, Appendix G- 
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NON-RESPONSE BIAS ADJUSTMENT 
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Example of, pg 73, pg 61 
Explanation of, pp 60-<= 
Importance of, pg 74 
Potential Problems in MIS, pg 64 
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(See also PRE-SURVEY LETTER) 
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ORIENTATION GUIDE, INTERVIEWER, SAMPLE, 
Exhibit 2.14, pp 62-65 

OVERSAMPLING, pg 26 

OVERVIEW OF FOLLOW-UP AND VALIDATION, 
SECTION 1, pp 1-27 
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PARTICIPANT, DEFINITION OF, Appendix A, pg 5 & 
Appendix B, pg 4 
PAYING RESPONDENTS, PROBLEMS OF, pp 23-24 
PERFORMANCE MANAGEMENT 
Feedback Mechanisms, pg 65 
Calculate Using Program and Reporting Years, 
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Employment Rate at Follow-Up, Formula, pg 70 
PREPARING PARTICIPANTS FOR FOLLOW-UP AND 
MOTIVATING PARTICIPATION, pp 38-41, 
pp 25-26 

PRETESTING, QUESTIONNAIRE, pg 7 

PRIVACY CONCERNS, pp 31-32 
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QUALITY CONTROL, pp 88-89, pp 51-52 
Benefits of Centratized, pg 18 
QUESTIONNAIRE DESIGN, pp 4-7 
Branching, pg 6 
Form Follows Function, pp 11-12 
Order Effeci, pg 5 
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Appendix C, (8 Pages), pg 12 

QUESTIONS, CLARIFYING AND PROBING, pg 51 
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Appendix D (5 Pages) 
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